1 Introduction

The discovery of a scalar particle at the LHC [1, 2] has provided important insight into the mechanism of electroweak symmetry breaking. Experimental studies of the new particle [37] demonstrate consistency with the standard model (SM) Higgs boson [813]. However, it remains possible that the discovered particle is part of an extended scalar sector, a scenario that is favoured by a number of theoretical arguments [14, 15].

The minimal supersymmetric standard model (MSSM) [1620] is the simplest extension of the SM that includes supersymmetry. The MSSM requires two Higgs doublets of opposite hypercharge. Assuming that CP symmetry is conserved, this results in one CP-odd (A) and two CP-even (h, H) neutral Higgs bosons and two charged Higgs bosons (\(H^{\pm }\)). At tree level, the properties of the Higgs sector in the MSSM depend on only two non-SM parameters, which can be chosen to be the mass of the CP-odd Higgs boson, \(m_A\), and the ratio of the vacuum expectation values of the two doublets, \(\tan \beta \). Beyond tree level, additional parameters affect the Higgs sector, the choice of which defines various MSSM benchmark scenarios. In some scenarios, such as \(m_{h}^{\text {mod}+}\) [21], the top-squark mixing parameter is chosen such that the mass of the lightest CP-even Higgs boson, \(m_h\), is close to the measured mass of the Higgs boson that was discovered at the LHC. A different approach is employed in the hMSSM scenario [22, 23] in which the value of \(m_h\) can be used, with certain assumptions, to predict the remaining masses and couplings of the MSSM Higgs bosons without explicit reference to the soft supersymmetry-breaking parameters. The couplings of the MSSM heavy Higgs bosons to down-type fermions are enhanced with respect to the SM for large \(\tan \beta \) values, resulting in increased branching fractions to \(\tau \) leptons and b-quarks,Footnote 1 as well as a higher cross section for Higgs boson production in association with b-quarks. This has motivated a variety of searches for a scalar boson in \(\tau \tau \) and bb final states at LEP [24], the Tevatron [2527] and the LHC [2832].

Heavy \(Z^\prime \) gauge bosons appear in several models [3337] and are a common extension of the SM [38]. Such \(Z^\prime \) bosons can appear in theories extending the electroweak gauge group, where lepton universality is typically conserved. A frequently used benchmark is the sequential standard model (SSM) [39], which contains a single additional \(Z^\prime \) boson with the same couplings as the SM Z boson. Some models offering an explanation for the high mass of the top-quark, predict instead that such bosons couple preferentially to third-generation fermions [4043]. A model predicting additional weak gauge bosons \(Z^\prime \) and \(W^\prime \) coupling preferentially to third-generation fermions is the strong flavour model (SFM) [41, 43].

Fig. 1
figure 1

Lowest-order Feynman diagrams for a gluon–gluon fusion and b-associated production in the b four-flavour and c five-flavour schemes of a neutral MSSM Higgs boson. Feynman diagram for Drell–Yan production of a \(Z^\prime \) boson at lowest order (d)

Direct searches for high-mass resonances decaying to \(\tau \tau \) have been performed by the ATLAS [44] and CMS [45] collaborations using 5 \(\mathrm{fb}^{-1}\) of integrated luminosity at \(\sqrt{s}=7~~{\mathrm {TeV}}\). ATLAS [46] updated the search with 20 \(\mathrm{fb}^{-1}\) of integrated luminosity at \(\sqrt{s}=8~~{\mathrm {TeV}}\). Indirect limits on \(Z^\prime \) bosons with non-universal flavour couplings have been set based on measurements from LEP [47].

This paper presents the results of a search for neutral MSSM Higgs bosons as well as high-mass \(Z^\prime \) resonances in the \(\tau \tau \) decay mode using 3.2 fb\(^{-1}\) of proton–proton collision data collected with the ATLAS detector [48] in 2015 at a centre-of-mass energy of 13  \({\mathrm {TeV}}\). The search is performed for the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) and \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) decay modes, where \(\tau _{\mathrm {lep}}\) represents the decay of a \(\tau \) lepton to an electron or a muon and neutrinos and \(\tau _{\mathrm {had}}\) represents the decay to one or more hadrons and a neutrino. The search considers narrow resonances in the mass range of 0.2–1.2  \({\mathrm {TeV}}\) and \(\tan \beta \) range of 1–60 for the MSSM Higgs bosons. For the \(Z^\prime \) boson search, the mass range of 0.5–2.5  \({\mathrm {TeV}}\) is considered. Higgs boson production through gluon–gluon fusion and in association with b-quarks is considered (Fig. 1a–1c), with the latter mode dominating for high \(\tan \beta \) values. Hence both the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) and \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channels are split into b-tag and b-veto categories, based on the presence or absence of jets originating from b-quarks in the final state. Since a \(Z^\prime \) boson is expected to be predominantly produced via a Drell–Yan process (Fig. 1d), there is little gain in splitting the data into b-tag and b-veto categories. Hence, the \(Z^\prime \) analysis uses an inclusive selection instead.

2 Data sample and Monte Carlo simulation

The ATLAS detector [48] at the LHC consists of an inner tracking detector with a coverage in pseudorapidityFootnote 2 up to \(|\eta |~=~2.5\) surrounded by a thin superconducting solenoid providing a 2 T axial magnetic field, electromagnetic and hadronic calorimeters extending up to \(|\eta |~=~4.9\) and a muon spectrometer covering \(|\eta |~<~2.7\). A new innermost layer was added to the pixel tracking detector after the end Run-1 at a radial distance of 3.3 cm from the beam line [49, 50]. The ATLAS trigger system consists of a hardware-based first level trigger, followed by a software-based high-level trigger (HLT). The integrated luminosity used in this search, considering the data-taking periods of 2015 in which all relevant detector subsystems were operational, is 3.2 fb\(^{-1}\). The luminosity measurement and its uncertainty are derived following a methodology similar to that detailed in Ref. [51], from a calibration of the luminosity scale using xy beam-separation scans performed in August 2015.

Simulated events with a heavy neutral MSSM Higgs boson produced via gluon–gluon fusion and in association with b-quarks are generated with the POWHEG-BOX v2 [5254] and MADGRAPH5_aMC@NLO 2.1.2 [55, 56] programs, respectively. The CT10 [57] and CT10nlo_nf4 [58] sets of parton distribution functions (PDFs) are used, respectively. PYTHIA 8.210 [59] with the AZNLO [60] (A14 [61]) set of tuned parameters, or “tune”, is used for the parton shower, underlying event and hadronization in the gluon–gluon fusion (b-associated) production. The production cross sections for the various MSSM scenarios are calculated using SusHi [62] for gluon fusion production [6375] and b-associated production in the five-flavour scheme [76]; b-associated production in the four-flavour scheme is calculated according to Refs. [77, 78]. The final b-associated production cross section is obtained by using the method in Ref. [79] to match the four-flavour and five-flavour scheme cross sections. The masses and the couplings of the Higgs bosons are computed with FeynHiggs [8084], whereas the branching fraction calculation follows the procedure described in Ref. [85]. In the case of the hMSSM scenario, the procedure described in Ref. [23] is followed for the production cross sections and HDECAY [86] is used for the branching fraction calculation.

The \(Z^\prime {}\) signals are simulated by reweighting a leading-order (LO) \(Z/\gamma ^{*}\rightarrow \tau \tau \) sample using the TauSpinner algorithm [8789] to account for spin effects in the \(\tau \) decays. The \(Z/\gamma ^{*}\rightarrow \tau \tau \) sample, enriched with high invariant mass events, is generated with PYTHIA 8.165 [90] using the NNPDF2.3LO PDF set [91] and the A14 tune for the underlying event. Interference between the \(Z^\prime \) signals and the SM \(Z/\gamma ^{*}\) production is not included.

The simulated backgrounds consist of the production of Z+jets, W+jets, \(t\bar{t}\) pairs, single top quarks and electroweak dibosons (WW / WZ / ZZ). These are modelled with several event generators as described below, while contributions from multi-jet production are estimated with data as described in Sect. 5.

Simulated samples of Z+jets events for the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) and \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channels and W+jets events for the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) channel are produced using POWHEG-BOX v2 interfaced to PYTHIA 8.186 with the AZNLO tune. In this sample, PHOTOS++ v3.52 [92, 93] is used for final-state QED radiation. A dedicated W+jets sample binned in \(p_{\text {T}} ^{W}\), produced using the SHERPA 2.1.1 generator [94], is used in the \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channel in order to enhance the number of events with high invariant mass. For this sample, matrix elements are calculated for up to two partons at next-to-leading order (NLO) and four partons at LO, merged with the SHERPA parton shower model using the ME+PS@NLO prescription [95]. Spin correlation effects between the W boson and its decay products are simulated with the TauSpinner program. All W/Z+jets samples use the CT10 PDF set and are normalized to the next-to-next-to-leading-order (NNLO) cross sections calculated using FEWZ [9698].

The POWHEG-BOX v2 program with the CT10 PDF set is used for the generation of \(t\bar{t}\) pairs and single top quarks in the Wt- and s-channels. Samples of t-channel single-top-quark events are produced with the POWHEG-BOX v1 generator employing the four-flavour scheme for the NLO matrix element calculations together with the fixed four-flavour scheme PDF set CT10f4; the top-quark decay is simulated with MadSpin [99]. For all samples of top-quark production, the spin correlations are preserved and the parton shower, fragmentation and underlying event are simulated using PYTHIA 6.428 [100] with the CTQ6L1 PDF set and the corresponding Perugia 2012 tune [101]. Final-state QED radiation is simulated using PHOTOS++ v3.52. The top-quark mass is set to 172.5 \(\text {GeV}\). The \(t\bar{t}\) production sample is normalized to the NNLO cross section, including soft-gluon resummation to next-to-next-to-leading-logarithm accuracy (Ref. [102] and references therein). The normalization of the single-top-quark event samples uses an approximate NNLO calculation from Refs. [103105].

Finally, diboson processes are simulated using the SHERPA 2.1.1 program with the CT10 PDF. They are calculated for up to one additional parton at NLO, depending on the process, and up to three additional partons at LO. The diboson samples use the NLO cross sections SHERPA calculates.

The simulation of b- and c-hadron decays for all samples, excluding those generated with SHERPA, uses EvtGen v1.2.0 [106]. All simulated samples include the effect of multiple proton-proton interactions in the same and neighbouring bunch crossings (“pile-up”) by overlaying simulated minimum-bias events on each generated signal or background event. These minimum-bias events are generated with PYTHIA 8.186 [90, 100], using the A2 tune [107] and the MSTW2008LO PDF [108]. Each sample is simulated using the full GEANT4 [109, 110] simulation of the ATLAS detector, with the exception of the b-associated MSSM Higgs boson signal, for which the ATLFAST-II [110, 111] fast simulation framework is used. Finally, the Monte Carlo (MC) samples are processed through the same reconstruction software as for the data.

3 Object reconstruction and identification

The primary vertex of each event is chosen as the proton?proton vertex candidate with the highest sum of the squared transverse momenta of all associated tracks. Electron candidates are reconstructed from energy deposits in the electromagnetic calorimeter associated with a charged-particle track measured in the inner detector. The final electron candidates are required to pass the “loose” likelihood-based identification selection [112, 113], to have a transverse energy \(E_{\text {T}} > 15\) \(\text {GeV}\) and to be in the fiducial volume of the inner detector, \(|\eta |<2.47\). The transition region between the barrel and end-cap calorimeters (\(1.37<|\eta |<1.52\)) is excluded.

Muon candidates are reconstructed from track segments in the muon spectrometer, matched with tracks found in the inner detector within \(|\eta |<2.5\). The tracks of the final muon candidates are refit using the complete track information from both detector systems and are required to have a transverse momentum \(p_{\text {T}} >15\) \(\text {GeV}\) and to pass the “loose” muon identification requirements [114].

Both the electrons and muons are required to pass a \(p_{\text {T}} \)-dependent isolation selection, which utilizes both calorimetric and tracking information, with an efficiency of 90 % (99 %) for transverse momentum of \(p_{\text {T}} =25~(60)\) \(\text {GeV}\). The isolation provides an efficiency that grows as a function of lepton \(p_{\text {T}} \), since the background from jets faking leptons becomes less important as the lepton \(p_{\text {T}} \) increases. The contributions from pile-up and the underlying event activity are corrected on an event-by-event basis using the ambient energy density technique [115].

Jets are reconstructed from topological clusters [116] in the calorimeter using the anti-\(k_t\) algorithm [117], with a radius parameter value \(R =0.4\). To reduce the effect of pile-up, a jet vertex tagger algorithm is used for jets with \(p_{\text {T}} < 50\) \(\text {GeV}\) and \(|\eta | < 2.4\). It employs a multivariate technique based on jet energy, vertexing and tracking variables to determine the likelihood that the jet originates from pile-up [118]. In order to identify jets containing b-hadrons (b-jets), a multivariate algorithm is used  [119, 120]. A working point that corresponds to an average efficiency of 70 % for b-jets in \(t\bar{t}\) simulated events is chosen. The misidentification rates for c-jets, \(\tau \)-jets and jets initiated by light quarks or gluons for the same working point and in the same sample of simulated \(t\bar{t}\) events are approximately 10, 4 and 0.2 % respectively.

Hadronic decays of \(\tau \) leptons are predominantly characterized by the presence of one or three charged particles, accompanied by a neutrino and possibly neutral pions. The reconstruction of the visible decay products, hereafter referred to as \(\tau _{\mathrm {had-vis}}\), starts with jets with \(p_{\text {T}} >10\) \(\text {GeV}\). The \(\tau _{\mathrm {had-vis}}\) candidate must have energy deposits in the calorimeters in the range \(|\eta | < 2.5\), with the transition region between the barrel and end-cap calorimeters excluded. Additionally, they must have \(p_{\text {T}} > 20~\text {GeV}\), one or three associated tracks and an electric charge of \(\pm 1\). A multivariate boosted decision tree (BDT) identification, based on calorimetric shower shape and track multiplicity of the \(\tau _{\mathrm {had-vis}}\) candidates, is used to reject backgrounds from jets. In this analysis, two \(\tau _{\mathrm {had-vis}}\) identification criteria are used: “loose” and “medium” with efficiencies measured in \(Z\rightarrow \tau \tau \) decays of about 60 % (50 %) and 55 % (40 %) for one-track (three-track) \(\tau _{\mathrm {had-vis}}\) candidates, respectively [121]. An additional dedicated likelihood-based veto is used to reduce the number of electrons misidentified as \(\tau _{\mathrm {had-vis}}\).

Signals in the detector can be used in more than one reconstructed object. Objects that have a geometric overlap are removed according to the following priorities:

  • Jets within a \(\Delta R = 0.2\) cone around a selected \(\tau _{\mathrm {had-vis}}\) are excluded.

  • Jets within a \(\Delta R = 0.4\) cone around an electron or muon are excluded.

  • Any \(\tau _{\mathrm {had-vis}}\) within a \(\Delta R = 0.2\) cone around an electron or muon is excluded.

  • Electrons within a \(\Delta R\) = 0.2 cone around a muon are excluded.

The missing transverse momentum (\(E_{\text {T}}^{\text {miss}} \)) is calculated as the modulus of the negative vectorial sum of the \(\mathbf {p_{\text {T}}}\) of all fully reconstructed and calibrated jets and leptons [122]. This procedure includes a “soft term”, which is calculated based on the inner-detector tracks originating from the primary vertex that are not associated to reconstructed objects.

4 Search channels

4.1 \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) channel

Events in the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) channel are recorded using single-muon triggers and a logical-OR combination of single-electron triggers. Single-electron triggers with \(p_{\text {T}} \) thresholds of \(24~\text {GeV}\), \(60~\text {GeV}\) and \(120~\text {GeV}\) are used for the \(\tau _e\tau _{\mathrm {had}}\) channel. For the \(\tau _\mu \tau _{\mathrm {had}}\) channel, a single-muon trigger with a \(p_{\text {T}} \) threshold of \(50~\text {GeV}\) is used if the muon \(p_{\text {T}} \) is larger than \(55~\text {GeV}\) and a single-muon trigger with a \(p_{\text {T}} \) threshold of \(20~\text {GeV}\) is used otherwise. The triggers impose electron and muon quality requirements which are tighter for the triggers with lower \(p_{\text {T}} \) thresholds.

Events must have at least one identified \(\tau _{\mathrm {had-vis}}\) candidate and either one electron or one muon candidate which is geometrically matched to the HLT object that triggered the event. Events with more than one electron or muon fulfilling the criteria described in Sect. 3 are rejected in order to reduce the backgrounds from \(Z/\gamma ^{*}\rightarrow \ell \ell \) production, where \(\ell = e\), \(\mu \). The selected lepton must have a transverse momentum \(p_{\text {T}} >30\,\text {GeV}\) and pass the “medium” identification requirement.

The \(\tau _{\mathrm {had-vis}}\) candidate is required to have \(p_{\text {T}} > 25~\text {GeV}\), pass the “medium” BDT-based identification requirement and lie in the range \(|\eta |<2.3\). The latter requirement is motivated by a larger rate of electrons misidentified as \(\tau _{\mathrm {had-vis}}\) candidates at higher \(|\eta |\) values: the rate is above 10 % for \(|\eta |>2.3\), while it ranges from 0.5 to 3 % for lower \(|\eta |\) values. If there is more than one \(\tau _{\mathrm {had-vis}}\) candidate, the candidate with the highest \(p_{\text {T}} \) is selected and the others are treated as jets. Finally, the identified lepton and the \(\tau _{\mathrm {had-vis}}\) are required to have opposite electric charge.

Subsequently, the following selection requirements are applied:

  • \(\Delta \phi (\tau _{\mathrm {had-vis}}, \ell ) > 2.4\).

  • \(m_\mathrm{T}(\ell ,E_{\text {T}}^{\text {miss}}) \equiv \sqrt{2p_{\text {T}} (\ell )E_{\text {T}}^{\text {miss}} \big [1-\cos \Delta \phi (\ell ,E_{\text {T}}^{\text {miss}})\big ]} < 40~\text {GeV}\).

  • For the \(\tau _{e}\tau _{\mathrm {had}}\) channel, events are vetoed if the invariant mass of the electron and the visible \(\tau \) lepton decay products is in the range \(80<m_{\text {vis}}(e,\tau _{\mathrm {had-vis}})<110~\text {GeV}\).

The requirement on \(\Delta \phi (\tau _{\mathrm {had-vis}}, \ell )\) gives an overall reduction of SM backgrounds with little signal loss. The requirement on \(m_\mathrm{T}(\ell ,E_{\text {T}}^{\text {miss}})\), the distribution of which is shown in Fig. 2a, serves to remove events that originate from processes containing a W boson: in signal events, the missing transverse momentum is usually in the same direction as the \(\tau _{\mathrm {lep}}\), resulting in a low value of \(m_\mathrm{T}(\ell ,E_{\text {T}}^{\text {miss}})\). The requirement on \(m_{\text {vis}}(e,\tau _{\mathrm {had-vis}})\) reduces the contribution of \(Z\rightarrow ee\) decays, where an electron is misidentified as a \(\tau _{\mathrm {had-vis}}\) candidate. These selection criteria define the inclusive \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) selection.

4.2 \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channel

Events in the \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channel are selected by a trigger that requires a single \(\tau _{\mathrm {had-vis}}\) satisfying the “medium” \(\tau _{\mathrm {had-vis}}\) identification criterion with \(p_{\text {T}} >80\) \(\text {GeV}\). The leading \(\tau _{\mathrm {had-vis}}\) candidate in \(p_{\text {T}} \) must geometrically match the HLT object. A \(p_{\text {T}} \) requirement is applied to the leading \(\tau _{\mathrm {had-vis}}\) candidate, \(p_{\text {T}} >110\) \(\text {GeV}\), and to the sub-leading \(\tau _{\mathrm {had-vis}}\) candidate, \(p_{\text {T}} >55\) \(\text {GeV}\). Furthermore, the leading (sub-leading) \(\tau _{\mathrm {had-vis}}\) candidate has to satisfy the “medium” (“loose”) \(\tau _{\mathrm {had-vis}}\) identification criterion. Events with electrons or muons fulfilling the loose selection criteria described in Sect. 3 (with the exception of the isolation requirement) are vetoed to reduce electroweak background processes and guarantee orthogonality with the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) channel.

The leading and sub-leading \(\tau _{\mathrm {had-vis}}\) candidates must have opposite electric charge and have a back-to-back topology in the transverse plane, \(\Delta \phi (\tau _{\mathrm {had-vis},1}, \tau _{\mathrm {had-vis},2})>2.7\). The distribution of \(\Delta \phi (\tau _{\mathrm {had-vis},1}, \tau _{\mathrm {had-vis},2})\) before this requirement is shown in Fig. 2b. This selection defines the inclusive \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) selection.

Fig. 2
figure 2

The distributions of a \(m_\mathrm{T}(\ell ,E_{\text {T}}^{\text {miss}})\) in the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) channel and b \(\Delta \phi (\tau _{\mathrm {had-vis},1}, \tau _{\mathrm {had-vis},2})\) for the \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channel for the inclusive selection with the criterion for the variable displayed removed. The label “Others” in b refers to contributions due to diboson, \(Z(\rightarrow \ell \ell )\)+jets and \(W(\rightarrow \ell \nu )\)+jets production. Bins have a varying size and overflows are included in the last bin of the distribution on the left

4.3 Event categories

In the search for \(Z^\prime \) bosons, the event selections described in Sects. 4.1 and 4.2 result in a signal selection efficiencyFootnote 3 varying between 0.8 % (2.0 %) at \(m_{Z^\prime }=500~ \text {GeV}\) and 3.4 % (3.8 %) at \(m_{Z^\prime }=2.5 ~{\mathrm {TeV}}\) for the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) (\(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\)) channel. In the search for the H / A bosons, events satisfying the inclusive selection are categorized to exploit the two different signal production modes as follows:

  • b-veto: no b-tag jets in the event,

  • b-tag: at least one b-tag jet in the event.

In the \(b\)-veto category, the H / A signal selection efficiency varies between 2 % at \(m_A=200\,\text {GeV}\) and 7 % at \(m_A=1.2\,~{\mathrm {TeV}}\) for the gluon–gluon fusion production mode in the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) channel, and from 0.1 to 15 % in the \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channel for the same mass range. In the \(b\text {-tagging}\) category, the signal selection efficiency varies from 0.5 % at \(m_A=200~\text {GeV}\) to  2% at \(m_A=1.2~~{\mathrm {TeV}}\) for the b-associated production mode in the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) channel, and from 0.1 to 6 % in the \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channel.

4.4 Di-tau mass reconstruction

The di-tau mass reconstruction is critical in achieving good separation between signal and background. However, its reconstruction is challenging due to the presence of neutrinos from the \(\tau \) lepton decays. The mass reconstruction used for both the \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) and \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) channels is the total transverse mass, defined as

$$\begin{aligned} m_\mathrm{T}^{\text {tot}} = \sqrt{ m_\mathrm{T}^{2}(E_{\text {T}}^{\text {miss}}, \tau _1) + m_\mathrm{T}^{2}(E_{\text {T}}^{\text {miss}}, \tau _2) + m_\mathrm{T}^{2}(\tau _1, \tau _2)}, \end{aligned}$$
(1)

where \(m_\mathrm{T}(a, b)\) is defined as

$$\begin{aligned} m_\mathrm{T}(a, b) = \sqrt{2p_{\text {T}} (a) p_{\text {T}} (b) [1-\cos \Delta \phi (a, b)]} \end{aligned}$$
(2)

and \(\tau \) refers to the visible decay of the \(\tau \) lepton (\(\ell \) or \(\tau _{\mathrm {had-vis}}\)). More complex mass reconstruction techniques were investigated, but they did not improve the expected sensitivity.

5 Background estimation

The background processes can be categorized according to whether the electron/muon and/or the \(\tau _{\mathrm {had-vis}}\) are correctly identified. Backgrounds from processes with correctly identified \(\tau _{\mathrm {had-vis}}\), electrons and muons, or where the \(\tau _{\mathrm {had-vis}}\) is due to a misidentified electron/muon in the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) channel, are estimated from simulation. Data-driven techniques are used for processes where the \(\tau _{\mathrm {had-vis}}\), or both the lepton and \(\tau _{\mathrm {had-vis}}\) are misidentified. The background contributions originating from processes where only the lepton is misidentified are found to be negligible.

Table 1 Description of the control regions used in the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) and \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channels

5.1 \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) background estimate

The main backgrounds in the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) channel arise from \(Z \rightarrow \tau \tau \) production, followed by processes with a misidentified \(\tau _{\mathrm {had-vis}}\) in the b-veto category and \(t\bar{t}\) production, with either a true \(\tau \) lepton or a jet misidentified as a \(\tau _{\mathrm {had-vis}}\), in the b-tag category.

Background processes where the \(\tau _{\mathrm {had}}\) candidate, or both the lepton and \(\tau _{\mathrm {had}}\) candidates, arise from misidentified jets are dominated by W+jets (\(t\bar{t}\)) and multi-jet processes, for the b-veto (b-tag) category. A data-driven “fake factor” (FF) technique is used to estimate the contribution of these processes to the signal region. The fake factors are derived separately for the b-veto and b-tag categories using fake factor control regions (see Table 1) dominated by a particular background process (Pr), and are defined as:

$$\begin{aligned} \text {FF}(\text {Pr}) = \frac{N(\text {nominal} \tau _{\mathrm {had-vis}}\text {ID}, \text {Pr})}{N(\text {anti-} \tau _{\mathrm {had-vis}}\text {ID}, \text {Pr})}, \end{aligned}$$
(3)

where \(N(\text {nominal} \) \(\tau _{\mathrm {had-vis}}\) \( \text {ID}, \text {Pr})\) is the number of \(\tau _{\mathrm {had-vis}}\) candidates in data satisfying the “medium” \(\tau _{\mathrm {had-vis}}\) identification criterion and \(N(\text {anti-}\) \(\tau _{\mathrm {had-vis}}\) \( \text {ID}, \text {Pr})\) is the number of \(\tau _{\mathrm {had-vis}}\) candidates failing this criterion but meeting a loose requirement on the BDT score. The latter requirement defines the “anti-\(\tau _{\mathrm {had}}\)” sub-region, which selects the same kind of objects mimicking \(\tau _{\mathrm {had-vis}}\) candidates as those fulfilling the identification criteria. The true \(\tau _{\mathrm {had}}\) contamination in the fake factor control regions is subtracted using simulation. In all the control regions, the fake factors are parameterized as a function of the transverse momentum and number of tracks of the reconstructed \(\tau _{\mathrm {had-vis}}\) object.

The fake factor for W+jets and \(t\bar{t}\) backgrounds, FF(W+jets\(/t\bar{t})\), is measured in a fake factor control region that is identical to the signal region, except that the \(m_\mathrm{T}(\ell ,E_{\text {T}}^{\text {miss}})\) selection criterion is reversed to \(m_\mathrm{T}(\ell ,E_{\text {T}}^{\text {miss}})> 60\,(70)~\text {GeV}\) for the \(\tau _{\mu }\tau _{\mathrm {had}}\) (\(\tau _{e}\tau _{\mathrm {had}}\)) channel. The purity of the W+jets background in the b-veto category of the control region is about \( 95\%\), while in the b-tag category both the W+jets and \(t\bar{t}\) processes contribute. The fake factor value for the b-tag category was found to be compatible with the value corresponding to the b-veto category. To improve the statistical precision, the fake factor measured in a control region without requirements on the number of b-tags is used for the b-tag category. The same fake factor is used in the search for the \(Z^\prime \) boson. For multi-jet events (MJ), the fake factor FF\((\text {MJ})\) is measured in a fake factor control region defined by inverting the isolation requirement on the electron or muon. The purity of multi-jet events in this control region exceeds 99 %. The fake factors are derived separately for the b-veto and b-tag categories by requiring no b-tag and at least one b-tag, respectively. For the \(Z^{\prime }\) analysis, no b-tag requirement is considered.

The shapes and normalization of background contributions in the signal region are then estimated by applying these fake factors to events that pass the anti-\(\tau _{\mathrm {had}}\) region selection but otherwise satisfy all signal region requirements. In this analysis, the fake factors are combined and weighted by the predicted contribution of each background process to the anti-\(\tau _{\mathrm {had}}\) region:

$$\begin{aligned} \text {FF}(\text {comb}) = \text {FF}(W+\text {jets}/t\bar{t}) \times r_{W/t\bar{t}} + \text {FF}(\text {MJ}) \times r_{\text {MJ}}, \end{aligned}$$
(4)

where \(r_{\text {MJ}}\) denotes the fraction of multi-jet events in the anti-\(\tau _{\mathrm {had}}\) region and \(r_{W/t\bar{t}} = 1 - r_{\text {MJ}}\). This neglects the differences between the fake factors for W+jets/\(t\bar{t}\) and other processes, such as Z production. The parameter \(r_{\text {MJ}}\) is estimated, separately for the b-veto and b-tag categories, in two steps using a data-driven method. First, the rates at which jets are misidentified as electrons or muons are measured from the ratio of leptons passing and failing the lepton isolation requirement in a region enriched in multi-jet events. This multi-jet control region is defined in Table 1. The predicted multi-jet rate is then applied to events in the anti-\(\tau _{\mathrm {had}}\) sub-region that also fail the lepton isolation, in order to calculate \(r_{\text {MJ}}\) as a function of \(\tau _{\mathrm {had-vis}}\) \(p_{\text {T}} \) separately for the \(\tau _e\tau _{\mathrm {had}}\) and \(\tau _\mu \tau _{\mathrm {had}}\) channels. When the fake factor is applied to the anti-\(\tau _{\mathrm {had}}\) sub-region events, the contributions from correctly identified \(\tau _{\mathrm {had-vis}}\) and from electrons and muons misidentified as \(\tau _{\mathrm {had-vis}}\) candidates are subtracted using the default MC simulation described in Sect. 2.

Background processes where the electron or the muon is identified as a \(\tau _{\mathrm {had-vis}}\) object are modelled using simulation. The main source of such backgrounds is \(Z (\rightarrow ee)\)+jets events in the \(\tau _{e}\tau _{\mathrm {had}}\) channel, which are reduced using the \(m_{\text {vis}}(e,\tau _{\mathrm {had}})\) mass-window veto described in Sect. 4.1. To account for mismodelling of electrons misidentified as \(\tau _{\mathrm {had-vis}}\) objects in \(Z\rightarrow ee\)+jets events, the simulation is corrected as a function of the lepton \(\eta \) using data control regions defined by reversing the mass-window criterion, as listed in Table 1. The correction amounts to 15 % for three-track \(\tau _{\mathrm {had-vis}}\), while for the one-track \(\tau _{\mathrm {had-vis}}\) objects the correction varies from 20 % in the barrel region to up to 200 % in the end-cap region.

The \(m_\mathrm{T}^{\text {tot}}\) distributions in the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) channel are shown in Fig. 3a, b for the W+jets control region and the \(t\bar{t}\) validation region, respectively. The latter is identical to the b-tag category definition, except for the \(m_\mathrm{T}(\ell ,E_{\text {T}}^{\text {miss}})\) requirement, which is reversed to \(m_\mathrm{T}(\ell ,E_{\text {T}}^{\text {miss}}) > 100~\text {GeV}\).

Fig. 3
figure 3

The distributions of \(m_\mathrm{T}^{\text {tot}}\) in a the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) channel W+jets control region, b the \(t\bar{t}\) validation region of the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) channel, c the \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channel b-veto same-sign validation region and d the \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channel b-tag same-sign validation region. The various control and validation regions are defined in Table 1. The data are compared to the background prediction and a hypothetical MSSM \(H/A\rightarrow \tau \tau \) signal (\(m_A = 500~\text {GeV}\) and \(\tan \beta = 20\)). The Monte Carlo statistics of the signal is limited in the background-dominated regions. The label “Others” in c and d refers to contributions due to diboson, \(Z(\rightarrow \ell \ell )\)+jets and \(W(\rightarrow \ell \nu )\)+jets production. The background uncertainty includes statistical and systematic uncertainties. The bins have a varying size and overflows are included in the last bin of the distributions

5.2 \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) background estimate

The dominant background process for the \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channel is multi-jet production, the cross section of which is several orders of magnitude higher than that of the signal processes. Despite the large suppression of this background thanks to the event selection, a sizeable contribution of events with two jets misidentified as \(\tau _{\mathrm {had-vis}}\) candidates remains. A fake factor technique is used to normalize and model this background. Fake factors parameterized as a function of \(p_{\text {T}} (\tau _{\mathrm {had-vis}})\) and the number of tracks are derived from a control region enriched with multi-jet events, described in Table 1. The factors are derived independently for the b-tag and b-veto categories in the search for H / A bosons, and inclusively in the search for \(Z^\prime \) bosons. They are then applied to data events where the leading \(\tau _{\mathrm {had-vis}}\) has passed the \(\tau _{\mathrm {had-vis}}\) identification requirement in the signal region, while the sub-leading \(\tau _{\mathrm {had-vis}}\) candidate has passed only a loose requirement on the BDT score. The contributions from non-multi-jet production processes are subtracted using simulation.

For \(t\bar{t}\) and W+jets events, along with other simulated background processes, the probability of a jet being misidentified as a \(\tau _{\mathrm {had-vis}}\) is modelled with a “fake-rate” technique. The rates of jets being misidentified as a \(\tau _{\mathrm {had-vis}}\) are measured from data as a function of the transverse momentum and number of tracks of the reconstructed \(\tau _{\mathrm {had-vis}}\). The fake-rate control regions are described in Table 1 and are enriched in \(W(\rightarrow \mu \nu )\)+jets for the b-veto category and \(t\bar{t}\) events for the b-tag category,

The fake rate is then applied to the simulated events as a weight for each of the reconstructed \(\tau _{\mathrm {had-vis}}\) that does not match geometrically a true \(\tau \) lepton. Fake rates derived in the fake-rate control region of the b-tag category are used for simulated \(t\bar{t}\) and single top quark events, while fake rates obtained in the b-veto control region are applied to the remaining processes. An additional weight is applied to \(W\rightarrow \tau \nu \)+jets events as a function of \(m_\mathrm{T}^\mathrm{tot}\), to improve the modelling of the kinematics of the W+jets simulated events. The reweighting function is derived by fitting the ratio of the data to the simulation for the \(W(\rightarrow \mu \nu )\mathrm{+jets}\) process in an additional \(W\rightarrow \mu \nu \) control region, defined in analogy with the inclusive signal selection and described in Table 1.

A same-sign validation region, enriched with events where at least one jet is misidentified as a \(\tau _{\mathrm {had-vis}}\), is obtained by inverting the opposite-sign requirement of the two \(\tau _{\mathrm {had-vis}}\) candidates. Distributions of \(m_\mathrm{T}^{\text {tot}}\) in the \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channel same-sign validation region are shown in Fig. 3c, d. The good performance multi-jet background estimation method is demonstrated by the agreement of the data with the background prediction.

6 Systematic uncertainties

The signal efficiency and the background estimates are affected by uncertainties associated with the detector simulation, the signal modelling and the data-driven background determination.

The integrated luminosity measurement has an uncertainty of 5 % and is used for all MC samples. Uncertainties related to the detector are included for all signal and backgrounds that are estimated using simulated samples. These uncertainties are also taken into account for simulated samples that are used in the derivation of data-driven background estimates. All instrumental systematic uncertainties arising from the reconstruction, identification and energy scale of \(\tau _{\mathrm {had-vis}}\) candidates, electrons, muons, (b-)jets and the soft term of the \(E_{\text {T}}^{\text {miss}}\) measurement are considered. The effect of the energy scale uncertainties on the objects is propagated to the \(E_{\text {T}}^{\text {miss}}\) determination. The electron, muon, jet and \(E_{\text {T}}^{\text {miss}} \) systematic uncertainties described above are found to have a very small effect.

Systematic uncertainties resulting from the data-driven background estimates are derived as follows. In the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) channel, the combined fake-factor method includes uncertainties in the W+jets\(/t\bar{t}\) fake factors, the multi-jet fake factors, and the \(r_{\text {MJ}}\) estimation. For the W+jets fake factors the main uncertainties arise from the dependence on \(\Delta \phi (\tau _{\mathrm {had-vis}},E_{\text {T}}^{\text {miss}})\), from the difference between the relative contributions of quark- and gluon-initiated jets faking \(\tau _{\mathrm {had-vis}}\) in the control region and the signal region, and from the contamination by backgrounds other than W+jets in the control region, which are estimated using simulation. The uncertainty is parameterized as a function of the anti-\(\tau _{\mathrm {had-vis}}\) \(p_{\text {T}}\) and amounts approximately to 17 % for jets misidentified as one-track \(\tau \) candidates and varies between 16 and 34 % for jets misidentified as three-track \(\tau \) candidates. Uncertainties related to non-W+jets events were studied and have no significant impact on the fake-factor determination. For the multi-jet fake factors and \(r_{\text {MJ}}\), the uncertainty is dominated by the number of data events in the control region and the subtraction of the remaining non-multi-jet backgrounds using simulation. Typical values of the total uncertainties for \(r_{\text {MJ}}\) are between 7 and 20 % and for the multi-jet fake factors between 10 and 20 %, depending on the channel and the \(\tau _{\mathrm {had-vis}}\) candidate \(p_{\text {T}} \). In addition, the effect on the background estimate due to the anti-\(\tau _{\mathrm {had}}\) region definition is examined. The loose \(\tau _{\mathrm {had-vis}}\) identification requirement used in the definition of this region is varied to estimate the corresponding uncertainty, which is 5 and 1 % in the \(\tau _e\tau _{\mathrm {had}}\) and the \(\tau _\mu \tau _{\mathrm {had}}\) channel, respectively.

In the \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channel, the uncertainty in the fake-factor measurement used for the multi-jet background estimation is obtained as the sum in quadrature of the statistical uncertainty of the measurement and the difference between the fake factors determined from same-sign and from opposite-sign events. The fake rates for jets misidentified as \(\tau _{\mathrm {had-vis}}\) are determined from data. The main systematic uncertainty arises from the statistical uncertainty of the fake-rate measurement and it ranges from 7 to 30 % as a function of the \(\tau _{\mathrm {had-vis}}\) \(p_{\text {T}} \). In the \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channel, the uncertainty in the parameters of the function used to reweight the \(W\rightarrow \tau \nu \)+jets background is propagated to the \(m_\mathrm{T}^\mathrm{tot}\) distribution, where its effect ranges from 5 to 20 %.

Theoretical cross-section uncertainties are considered for all backgrounds estimated using simulation. For Z+jets and diboson production, uncertainties of \(5\%\) and \(6\%\) are used, respectively, combining PDF\(+\alpha _{\text {S}} \) and scale variation uncertainties in quadrature. For \(t\bar{t}\) [102] and single top-quark [123, 124] production, a \(6\%\) uncertainty is assigned based on scale, PDF and top-quark mass uncertainties. Additional uncertainties related to initial- and final-state radiation modelling, tune and (for \(t\bar{t} \) only) the choice of the hdamp parameter value in POWHEG-BOX v2, which controls the amount of radiation produced by the parton shower, were also taken into account [125]. The uncertainty in the fragmentation model is evaluated by comparing \(t\bar{t} \) events generated with POWHEG-BOX v2 interfaced to either HERWIG++ [126] or PYTHIA6. The POWHEG+HERWIG++ and aMC@NLO+HERWIG++ generators are used to estimate the uncertainty in generating the hard scatter. The variation of the b-tag category acceptance for these uncertainties is from \(-10\) to \(+30\,\%\) (\(-33\) to \(+38\,\%\)) in the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) (\(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\)) channel.

Uncertainties related to signal modelling are discussed in the following. Uncertainties due to the factorization and renormalization scale choices are estimated from the effect on the signal acceptance of doubling or halving these factors either coherently or oppositely. Uncertainties due to the initial- and final-state radiation, as well as multiple parton interaction for the signal are also taken into account. These uncertainties are estimated from the PYTHIA8 A14 tune [61] for the b-associated production and the AZNLO PYTHIA8 tune [60] for the gluon–gluon fusion production. The envelope of the variations resulting from the use of the alternative PDFs in the PDF4LHC15\(\_\)nlo\(\_\)100 [127] set is used in order to estimate the PDF uncertainty for gluon–gluon fusion production. For the b-associated production uncertainty, a comparison among NNPDF30_nlo_as_0118_nf_4 [127], CT14nlo_NF4 [58], MSTW2008nlo68cl_nf4 [128] and CT10_nlo_nf4 [57] PDF sets is employed. Since no statistically significant effect on the shape of the reconstructed mass distribution is observed, each contribution is taken solely as a normalization uncertainty. The total uncertainty ranges between 15 and 25 %.

7 Results

The parameter of interest is the signal strength, \(\mu \). It is defined as the ratio of the observed to the predicted value of the cross section times branching fraction, where the prediction is evaluated for a particular MSSM or \(Z^\prime \) assumption. Hence, the value \(\mu =0\) corresponds to the absence of a signal, whereas the value \(\mu =1\) indicates the presence of a signal as predicted by the theoretical model under study. To estimate \(\mu \), a likelihood function constructed as the product of Poisson probability terms is used. Signal and background predictions depend on systematic uncertainties, which are parameterized as nuisance parameters and are constrained using Gaussian probability distributions. For the MSSM Higgs boson search a binned likelihood function is constructed in bins of the \(m_\mathrm{T}^\mathrm{tot}\) distributions, chosen to ensure sufficient background statistics in each bin. The search for a \(Z^\prime \) boson is a counting experiment, summing the number of events above a certain \(m_\mathrm{T}^{\text {tot}}\) threshold. The threshold is chosen for each \(Z^\prime \) mass hypothesis to maximize the expected significance and ranges from 400 \(\text {GeV}\) at low \(Z^\prime \) mass to 750 \(\text {GeV}\) at high \(Z^\prime \) mass. The asymptotic approximation is used with the test statistic \(\tilde{q}_\mu \) [129] to test the compatibility of the data with the assumed signal.

The number of observed \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) and \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) data events, along with the predicted event yields from background and signal processes, in the signal regions are shown in Table 2. The observed event yields are compatible with the expected event yield from SM processes, within uncertainties. The \(m_\mathrm{T}^{\text {tot}}\) mass distributions are shown in Fig. 4. The results from the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) and \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channels are combined to improve the sensitivity to H / A and \(Z^\prime \) boson production.

Table 2 Observed number of events and background predictions in the b-tag and b-veto categories for the \(\tau _{e}\tau _{\mathrm {had}}\), \(\tau _{\mu }\tau _{\mathrm {had}}\) and \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channels. The background predictions and uncertainties are obtained from the statistical procedure discussed in Sect. 7. In the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) channel, the processes other than “Jet \(\rightarrow \ell , \tau _{\mathrm {had-vis}}\) fakes” require a true hadronically decaying \(\tau \) lepton or an electron or muon misidentified as a \(\tau _{\mathrm {had-vis}}\). The expected signal yields for the \(m_{h}^{\text {mod}+}\) scenario are shown for comparison
Fig. 4
figure 4

The distribution of \(m_\mathrm{T}^{\text {tot}}\) for the b-veto category of the a \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) and b \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channels, the b-tag category of the c \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) and d \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channels, and the inclusive category of the e \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) and f \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channels. The label “Others” in b, d and f refers to contributions due to diboson, \(Z(\rightarrow \ell \ell )\)+jets and \(W(\rightarrow \ell \nu )\)+jets production. For the b-veto and b-tag categories, the binning displayed is that entering into the statistical fit discussed in Sect. 7, while the predictions and uncertainties for the background and signal processes are obtained from the fit under the hypothesis of no signal. The inclusive category distributions are shown before any statistical fit. Overflows are included in the last bin of the distributions

The fractional contributions of the most important sources of systematic uncertainty to the total uncertainty in the signal cross-section measurement are shown for two signal assumptions: Table 3(top) represents an MSSM Higgs boson hypothesis (\(m_A=500\) \(\text {GeV}\), \(\tan \beta =20\)) and Table 3(bottom) corresponds to an SSM \(Z^\prime \) boson hypothesis (\(m_{Z^\prime }=1.75\)  \({\mathrm {TeV}}\)). As shown in this table, the sensitivity of the search is limited by statistical uncertainties.

Table 3 Fractional impact of the most important sources of systematic uncertainty on the total uncertainty of the signal strength, for (top) the MSSM signal hypothesis of \(m_A=500\) \(\text {GeV}\), \(\tan \beta =20\), (bottom) the \(Z^\prime _\mathrm {SSM}\) signal hypothesis of \(m_{Z^\prime }=1.75\)  \({\mathrm {TeV}}\). For each source of uncertainty, \(F_{\pm } = \pm \frac{\sigma ^{2}_{\text {source}}}{\sigma ^{2}_{\text {total}}}\) is defined as the positive (negative) fractional contribution to the signal strength uncertainty

The data are found to be in good agreement with the predicted background yields and hence the results are given as exclusion limits. These are set using the modified frequentist method known as CL\(_{s}\) [130]. Observed and expected 95 % confidence level (CL) upper limits on the cross section times branching fraction for the production of a single scalar boson H / A decaying to \(\tau \tau \), as a function of the mass of the boson \(m_{H/A}\), are shown in Fig. 5a, b. The limits are calculated for both the gluon–gluon fusion and b-associated production modes, using a combination of the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) and \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channels and assuming the natural width of the boson to be negligible compared to the experimental resolution (as expected over the probed MSSM parameter space). The lowest excluded cross section times branching fraction values range from \(\sigma \times BR = 1.4~\mathrm {pb}\) at \(m_{H/A} = 200~\text {GeV}\) to \(\sigma \times BR = 0.025~\mathrm {pb}\) at \(m_{H/A} = 1.2~~{\mathrm {TeV}}\) for a scalar boson produced via gluon–gluon fusion. Similarly, for the b-associated production mechanism the lowest excluded values range is from \(\sigma \times BR = 1.6~\mathrm {pb}\) at \(m_{H/A} = 200\,\text {GeV}\) to \(\sigma \times BR = 0.028~\mathrm {pb}\) at \(m_{H/A} = 1.2\,~{\mathrm {TeV}}\).

Fig. 5
figure 5

The observed and expected 95 % CL upper limits on the production cross section times branching fraction of a scalar particle are shown for the combination of the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) and the \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channels. The production mechanism of \(H/A\rightarrow \tau \tau \) is assumed to be a gluon–gluon fusion or b b-associated production. For comparison, the expected limits for the individual channels, \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) and \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\), are shown as well. The observed and expected 95 % CL limits on \(\tan \beta \) as a function of \(m_A\) are shown in c for the MSSM \(m_h^{\text {mod+}}\) scenario and d for the hMSSM scenario. For comparison, the expected limits from the individual channels, \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) and \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\), are given in c, while the observed and expected limits from the ATLAS Run-1 analysis in Ref. [28] are shown in d. Dashed lines of constant \(m_h\) and \(m_H\) are shown in red and blue, respectively

The observed and expected 95 % CL limits on \(\tan \beta \) as a function of \(m_A\), for the combination of \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) and \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channels in the MSSM \(m_h^{\text {mod+}}\) and hMSSM scenarios are shown in Fig. 5c, d. The expected limit in the \(m_h^{\text {mod+}}\) scenario is compared to the expected limits from the individual \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) and \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channels. For the \(m_h^{\text {mod+}}\) figure, lines of constant \(m_h\) and \(m_H\) are shown. For the hMSSM scenario, the exclusion arising from the SM Higgs boson coupling measurements of Ref. [131] is also shown, in addition to the ATLAS Run-1 \(H/A\rightarrow \tau \tau \) search result of Ref. [28]. The \(\tan \beta \) constraints in the hMSSM scenario are stronger than those in the \(m_h^{\text {mod+}}\) scenario. This is due to the presence of low-mass neutralinos in the \(m_h^{\text {mod+}}\) scenario that reduce the \( H/A\rightarrow \tau \tau \) branching fraction and which are absent in the hMSSM scenario. In the hMSSM scenario, the most stringent constraints on \(\tan \beta \) for the combined search exclude \(\tan \beta > 7.1\) for \(m_A = 200~\text {GeV}\) and \(\tan \beta > 39\) for \(m_A = 1~~{\mathrm {TeV}}\) at the 95 % CL. In the MSSM \(m_h^{\text {mod+}}\) scenario, the 95 % CL upper limits exclude \(\tan \beta > 7.6\) for \(m_A = 200~\text {GeV}\) and \(\tan \beta > 47\) for \(m_A = 1~~{\mathrm {TeV}}\). The feature of the expected limits in the hMSSM scenario exclusion plot at around \(m_A = 350\) \(\text {GeV}\) is due to the behaviour of the branching ratio \(A\rightarrow \tau \tau \) close to the \(A\rightarrow t\bar{t} \) kinematic threshold. Some sensitivity of the search is also expected around \(\tan \beta \sim 1\), \(m_A\sim 200\) \(\text {GeV}\) due to the increase of the gluon–gluon fusion cross section induced by the increased coupling to the top quark.

In the search for the \(Z^\prime \) boson, the observed number of events in the signal regions of the \(\tau _{\mathrm {lep}}\tau _{\mathrm {had}}\) and \(\tau _{\mathrm {had}}\tau _{\mathrm {had}}\) channels are consistent with the SM predictions. The resulting 95% CL upper limits are set on the cross section times branching fraction as a function of the mass and shown in Fig. 6a. These results are interpreted in the context of the SSM and SFM in Fig. 6a, b, respectively. The resulting observed (expected) lower limit on the mass of the \(Z'_\mathrm{SSM}\) boson is \(1.90{}\) (\(1.84\))  \({\mathrm {TeV}}\). In the search for the \(Z^\prime _\mathrm {SFM}\) boson, results are presented as a function of \(\sin ^{2}\phi {}\), where \(\phi \) is the mixing angle between the two SU(2) gauge eigenstates of the model. Masses below 1.82–2.17  \({\mathrm {TeV}}\) are excluded in the range \(0.1< \sin ^{2}\phi {} < 0.5\), assuming no \(\mu -\tau \) mixing. For the value of \(\sin ^{2}\phi {}=0.03\), the lower limit on the mass of a \(Z^\prime _\mathrm {SFM}\) boson is 2.12  \({\mathrm {TeV}}\), extending the limits from previous direct and indirect searches by more than 200 \(\text {GeV}\).

Fig. 6
figure 6

The 95 % CL upper limit on the cross section times branching fraction for a \(Z'\rightarrow \tau \tau \) in a the Sequential Standard Model and 95% CL exclusion on b the SFM parameter space, overlaid with indirect limits at 95 % CL from fits to electroweak precision measurements [132], lepton flavour violation [133], CKM unitarity [134] and Z-pole measurements [41]

8 Conclusions

A search for neutral Higgs bosons of the minimal supersymmetric standard model (MSSM) and for a \(Z^\prime \) gauge boson decaying to a pair of \(\tau \) leptons is performed using a data sample corresponding to an integrated luminosity of 3.2 fb\(^{-1}\) from proton–proton collisions at \(\sqrt{s} = 13\)  \({\mathrm {TeV}}\) recorded by the ATLAS detector at the LHC. The search finds no indication of an excess over the expected background in the channels considered. Limits are set at the 95 % CL, which provide constraints in the MSSM parameter space. Model-independent upper limits on the production cross section times the \(\tau \tau \) branching fraction of a scalar boson versus its mass, in both the gluon–gluon fusion and b-associated production modes, are presented. The upper limits on the cross section times branching fraction range from \(1.4~(1.6)~\mathrm {pb}\) at \(m_{H/A} = 200\,\text {GeV}\) to \(0.025~(0.028)~\mathrm {pb}\) at \(m_{H/A} = 1.2\,~{\mathrm {TeV}}\) for a scalar boson produced via gluon–gluon fusion (b-associated production). In the context of the MSSM \(m_{h}^{\text {mod+}}\) scenario, the most stringent 95 % CL upper limit on \(\tan \beta \) for the combined search is \(\tan \beta < 7.6\) for \(m_A = 200~\text {GeV}\). This analysis extends the limits of the previous searches for the mass range \(m_A > 500~\text {GeV}\). The search for a \(Z^\prime \) boson is interpreted in the context of the sequential standard model (SSM) and the strong flavour model (SFM). Upper limits at the 95 % CL are set on the cross section times branching fraction as a function of the \(Z^\prime \) mass. The observed lower limit on the \(Z^\prime \) mass is \(1.90{}\)  \({\mathrm {TeV}}\) for a \(Z'_\mathrm{SSM}\) and ranges from 1.82 to 2.17  \({\mathrm {TeV}}\) as a function of the \(\sin ^{2}\phi {}\) parameter for a \(Z^\prime _\mathrm {SFM}\).