Search for triboson $W^{\pm }W^{\pm }W^{\mp }$ production in pp collisions at $\sqrt{s}=8$ $\text {TeV}$ with the ATLAS detector

Abstract

This paper reports a search for triboson $W^{\pm }W^{\pm }W^{\mp }$ production in two decay channels (${W^{\pm }W^{\pm }W^{\mp } \rightarrow \ell ^\pm \nu \ell ^\pm \nu \ell ^\mp \nu }$ and ${W^{\pm }W^{\pm }W^{\mp } \rightarrow \ell ^\pm \nu \ell ^\pm \nu jj}$ with $\ell =e, \mu$) in proton-proton collision data corresponding to an integrated luminosity of 20.3 $\mathrm{fb}^\mathrm{-1}$ at a centre-of-mass energy of 8 $\text {TeV}$ with the ATLAS detector at the Large Hadron Collider. Events with exactly three charged leptons, or two leptons with the same electric charge in association with two jets, are selected. The total number of events observed in data is consistent with the Standard Model (SM) predictions. The observed 95% confidence level upper limit on the SM $W^{\pm }W^{\pm }W^{\mp }$ production cross section is found to be 730 fb with an expected limit of 560 fb in the absence of SM $W^{\pm }W^{\pm }W^{\mp }$ production. Limits are also set on WWWW anomalous quartic gauge couplings.

1 Introduction

The triple gauge couplings (TGCs) and quartic gauge couplings (QGCs) that describe the strengths of the triple and quartic gauge boson self-interactions are completely determined by the non-Abelian nature of the electroweak SU(2)$_{\text {L}} \times$ U(1)$_{\text {Y}}$ gauge structure in the Standard Model (SM). These interactions contribute directly to diboson and triboson production at colliders. Studies of triboson production can test these interactions and any possible observed deviation from the theoretical prediction would provide hints of new physics at a higher energy scale. Compared with TGCs, QGCs are usually harder to study due to the, in general, smaller production cross sections of the relevant processes.

In the SM, charged QGC interactions (WWWW, WWZZ, $WWZ\gamma$ and $WW\gamma \gamma$) are allowed whereas neutral QGC interactions (ZZZZ, $ZZZ\gamma$, $ZZ\gamma \gamma$, $Z\gamma \gamma \gamma$ and $\gamma \gamma \gamma \gamma$) are forbidden. Searches have been performed by the LEP experiments for $WW\gamma \gamma$, $WWZ\gamma$, and $ZZ\gamma \gamma$ QGCs [1,2,3,4,5,6], by the Tevatron experiments for $WW\gamma \gamma$ [7], and by the LHC experiments for $WW\gamma \gamma$, $WWZ\gamma$, WWZZ, $ZZ\gamma \gamma$, $Z\gamma \gamma \gamma$, and WWWW QGCs [8,9,10,11,12,13,14,15,16,17].

Previous studies of WWWW QGC interactions [8, 16] used $W^\pm W^\pm$ vector-boson scattering events, whereas this paper presents the first search for WWWW QGC interactions via triboson $W^{\pm }W^{\pm }W^{\mp }$ production and sets the first limit on the total SM $W^{\pm }W^{\pm }W^{\mp }$ production cross-section using proton-proton (pp) collision data collected with the ATLAS detector and corresponding to an integrated luminosity of 20.3 $\mathrm{fb}^\mathrm{-1}$ [18] at a centre-of-mass energy of 8 $\text {TeV}$. Two decay channels, $W^{\pm }W^{\pm }W^{\mp } \rightarrow \ell ^\pm \nu \ell ^\pm \nu \ell ^\mp \nu$ and $W^{\pm }W^{\pm }W^{\mp } \rightarrow \ell ^\pm \nu \ell ^\pm \nu jj$ , with $\ell =e$ or $\mu$, are considered and are hereafter referred to simply as $\ell \nu \ell \nu \ell \nu$ and $\ell \nu \ell \nu jj$ channels, respectively.

2 The ATLAS detector

The ATLAS detector [19] is composed of an inner tracking detector (ID) surrounded by a thin superconducting solenoid providing a 2 T axial magnetic field, electromagnetic and hadronic calorimeters, and a muon spectrometer (MS). The ID consists of three subsystems: the pixel and silicon microstrip detectors that cover $|\eta |<2.5$ in pseudorapidity,¹ and the outer transition radiation tracker that has an acceptance range of $|\eta |<2.0$. The finely-segmented electromagnetic calorimeter is composed of lead absorbers with liquid argon (LAr) as the active material, spanning $|\eta |<3.2$. In the region $|\eta |<1.8$, a pre-sampler detector using a thin layer of LAr is used to correct for the energy loss by electrons and photons upstream of the calorimeter. The hadronic tile calorimeter ($|\eta |<1.7$) consists of steel absorbers and scintillating tiles and is located directly outside the envelope of the barrel electromagnetic calorimeter. The endcap hadronic calorimeters use LAr as active material, with copper as absorber material, while the forward calorimeters use LAr as active material, with copper absorber for the first layer, dedicated to electromagnetic measurements, and tungsten for other layers, dedicated to hadronic measurements. The MS is composed of three large superconducting air-core toroidal magnets, a system of three stations of tracking chambers in the range $|\eta |<2.7$, and a muon trigger system in the range $|\eta |< 2.4$. The precision muon momentum measurement is performed by monitored drift tubes everywhere except in the innermost layer for the range $|\eta | > 2.0$ where cathode strip chambers are used instead. The muon trigger system is composed of resistive plate chambers in the barrel region ($|\eta | < 1.05$) and thin gap chambers in the endcap region ($1.05< |\eta | < 2.4$).

The ATLAS trigger system has three distinct levels referred to as L1, L2, and the event filter. Each trigger level refines the decisions made at the previous level. The L1 trigger is implemented in hardware and uses a subset of detector information to reduce the event rate to a design value of at most 75 kHz. The L2 and event filter are software-based trigger levels and together reduce the event rate to about 400 Hz.

Events used were selected by single-lepton triggers with a transverse momentum, $p_{\text {T}}$ , threshold of 24 $\text {GeV}$ for both muons and electrons, along with an isolation requirement. The single-lepton triggers are complemented with triggers having a higher $p_{\text {T}}$ threshold (60 $\text {GeV}$ for electrons and 36 $\text {GeV}$ for muons) and no isolation requirement in order to increase the acceptance at high $p_{\text {T}}$.

3 Object reconstruction and event selection

Each event is required to have at least one primary vertex reconstructed from at least three tracks with $p_{\text {T}} >400$ $\text {MeV}$. If there are multiple primary vertices reconstructed in the event due to additional pp interactions (pile-up) in the same or a neighbouring bunch crossing, the vertex with the highest $\sum p_{\text {T}} ^2$, calculated using all associated tracks, is taken as the primary collision vertex. The mean number of interactions per bunch crossing in this data set is 20.7.

Electron candidates [20] are required to have $p_{\text {T}} >20$ $\text {GeV}$ and $|\eta | < 2.47$. Candidates within the transition region between the barrel and endcap calorimeters ($1.37< |\eta | < 1.52$) are rejected. In addition, they must satisfy the tight quality definition described in Ref. [21]. Muon candidates are reconstructed by combining tracks in the ID with tracks in the MS and have $p_{\text {T}} >20$ $\text {GeV}$ and $|\eta |<2.5$. The ID tracks associated with these muons must pass a number of quality requirements [22].

To ensure that lepton candidates originate from the primary vertex, a requirement is placed on the longitudinal impact parameter, $z_0$, multiplied by the sine of the track polar angle, $\theta$, such that the absolute value is smaller than 0.5 mm ($|z_0 \times \sin \theta |<0.5$ mm). A requirement is also placed on the transverse impact parameter, $d_0$, divided by its resolution ($\sigma _{d_0}$), such that $|d_0/\sigma _{d_0}|<3$. To suppress the contribution from hadronic jets which are misidentified as leptons, signal leptons are required to be isolated in both the ID and the calorimeter. The calorimeter isolation is defined as $E_{\text {T}} ^{\mathrm {Cone}X}/E_{\text {T}}$ whereas the ID isolation is defined as $p_{\text {T}} ^{\mathrm {Cone}X}/p_{\text {T}}$, where $E_{\text {T}} ^{\mathrm {Cone}X}$ ($p_{\text {T}} ^{\mathrm {Cone}X}$) is the transverse energy (momentum) deposited in the calorimeter (the scalar sum of the $p_{\text {T}}$ of tracks with $p_{\text {T}} >1$ $\text {GeV}$) within a cone of size $\Delta R=\sqrt{(\Delta \eta )^2 + (\Delta \phi )^2}=X$ around the lepton. The transverse momentum from the lepton itself is excluded in the calculations of $E_{\text {T}} ^{\mathrm {Cone}X}$ and $p_{\text {T}} ^{\mathrm {Cone}X}$. Different lepton isolation criteria are applied in the two channels to maximize the signal efficiency while suppressing the backgrounds. In the $\ell \nu \ell \nu \ell \nu$ channel, $E_{\text {T}} ^{\mathrm {Cone}0.2}/E_{\text {T}} <0.1$ and $p_{\text {T}} ^{\mathrm {Cone}0.2}/p_{\text {T}} <0.04$ are required for both the electrons and muons; in the $\ell \nu \ell \nu jj$ channel, $E_{\text {T}} ^{\mathrm {Cone}0.3}/E_{\text {T}} <0.14$ and $p_{\text {T}} ^{\mathrm {Cone}0.3}/p_{\text {T}} <0.06$ are required for electrons whereas $E_{\text {T}} ^{\mathrm {Cone}0.3}/E_{\text {T}} <0.07$ and $p_{\text {T}} ^{\mathrm {Cone}0.3}/p_{\text {T}} <0.07$ are required for muons.

Jets are reconstructed from clusters of energy in the calorimeter using the anti-$k_t$ algorithm [23] with radius parameter $R = 0.4$. Jet energies are calibrated using energy- and $\eta$-dependent correction factors derived using Monte Carlo (MC) simulation and validated by studies of collision data [24]. For jets with $p_{\text {T}} <50$ $\text {GeV}$ and $|\eta |<2.4$, at least 50% of the summed scalar $p_{\text {T}}$  of the tracks within a cone of size $\Delta R=0.4$ around the jet axis must originate from the primary vertex. This requirement reduces the number of jet candidates originating from pile-up vertices. Jets containing b-hadrons (“b-jets”) with $|\eta | < 2.5$ and $p_{\text {T}} >25~\text {GeV}$ are identified using the impact parameter significance of tracks in the jet and secondary vertices reconstructed from these tracks [25, 26]. In the $\ell \nu \ell \nu \ell \nu$ and $\ell \nu \ell \nu jj$ channels, the efficiency of the b-tagging algorithm used is 85 and 70%, respectively.

The measurement of the two-dimensional missing transverse momentum vector, $\vec {p}_{\text {T}}^{\text {~miss}}$, is based on the measurement of all topological clusters in the calorimeter and muon tracks reconstructed in the ID and MS [27]. Calorimeter cells associated with reconstructed objects, such as electrons, photons, hadronically decaying $\tau$ leptons, and jets, are calibrated at their own energy scale, whereas calorimeter cells not associated with any object are calibrated at the electromagnetic energy scale and taken into account as a so-called “soft term” in the calculation of $\vec {p}_{\text {T}}^{\text {~miss}}$. The magnitude of the missing transverse momentum vector is referred to as the missing transverse energy, $E_{\text {T}}^{\text {miss}} = |\vec {p}_{\text {T}}^{\text {~miss}} |$.

The experimental signature of the $\ell \nu \ell \nu \ell \nu$ channel is the presence of three charged leptons and $E_{\text {T}}^{\text {miss}}$ . The signature of the $\ell \nu \ell \nu jj$ channel is the presence of two same-charge leptons, $E_{\text {T}}^{\text {miss}}$, and two jets with an invariant mass close to 80 $\text {GeV}$. The selection requirements used to define the signal regions described in the following are obtained from a multi-dimensional optimization to maximize the sensitivity to the $W^{\pm }W^{\pm }W^{\mp }$ process and to reduce the contributions from SM background processes.

To select $\ell \nu \ell \nu \ell \nu$ candidates, events are required to have exactly three charged leptons with $p_{\text {T}} >20$ $\text {GeV}$, at most one jet with $p_{\text {T}} >25$ $\text {GeV}$ and $|\eta |<4.5$, and no identified b-jets. In addition, the absolute value of the azimuthal angle between the trilepton system and the $\vec {p}_{\text {T}}^{\text {~miss}}$, $|\phi ^{3\ell } - \phi ^{\vec {p}_{\text {T}}^{\text {~miss}}}|$, is required to be above 2.5. Eight different final states with equal production probability are considered based on the flavour and the charge of the leptons, namely $e^\pm e^\pm e^\mp$, $e^{\pm }e^{\mp }\mu ^{\pm }$, $e^{\pm }e^{\mp }\mu ^{\mp }$, $e^\pm e^\pm \mu ^\mp$, $\mu ^{\pm }\mu ^{\mp }e^{\pm }$, $\mu ^{\pm }\mu ^{\mp }e^{\mp }$, $\mu ^\pm \mu ^\pm e^\mp$, and $\mu ^\pm \mu ^\pm \mu ^\mp$. Three separate signal regions are defined based on the number of same-flavour opposite-sign (SFOS) lepton pairs in the event: 0 SFOS ($e^\pm e^\pm \mu ^\mp$ and $\mu ^\pm \mu ^\pm e^\mp$), 1 SFOS ($e^{\pm }e^{\mp }\mu ^{\pm }$, $e^{\pm }e^{\mp }\mu ^{\mp }$, $\mu ^{\pm }\mu ^{\mp }e^{\pm }$, and $\mu ^{\pm }\mu ^{\mp }e^{\mp }$), and 2 SFOS ($e^\pm e^\pm e^\mp$ and $\mu ^\pm \mu ^\pm \mu ^\mp$). In the 0-SFOS case, the invariant mass of the same-flavour lepton pair, $m_{\ell \ell }$, is required to be greater than 20 $\text {GeV}$. If there are at least two electrons in the event, the di-electron invariant mass, $m_{ee}$, is required to have $|m_{ee} - m_Z|>15$ $\text {GeV}$, where $m_Z$ is the pole mass of the Z boson [28]. No requirement is applied on the $E_{\text {T}}^{\text {miss}}$ variable, as it was found to not discriminate between signal and backgrounds. In the 1-SFOS case, the SFOS dilepton invariant mass, $m_\mathrm{SFOS}$, is required to be outside of the region $m_Z-35~\text {GeV}{}<m_{\text {SFOS}}<m_Z+20$ $\text {GeV}$. In addition, events are required to satisfy $E_{\text {T}}^{\text {miss}} >45$ $\text {GeV}$. Finally, in the 2-SFOS case, the SFOS dilepton invariant masses are required to have $|m_\mathrm{SFOS}-m_Z|>20$ $\text {GeV}$ while the $E_{\text {T}}^{\text {miss}}$  must be greater than 55 $\text {GeV}$. The selection criteria for $m_\mathrm{SFOS}$ and $E_{\text {T}}^{\text {miss}}$ are mainly used to reduce the contributions from the $Z+$jets and $WZ+$jets processes. Table 1 shows the kinematic selection criteria used for the $\ell \nu \ell \nu \ell \nu$ channel.

Table 1 Selection criteria for the $\ell \nu \ell \nu \ell \nu$ channel, split based on the number of SFOS lepton pairs: 0 SFOS, 1 SFOS, and 2 SFOS

To select $\ell \nu \ell \nu jj$ candidates, events are required to have exactly two leptons with the same electric charge, at least two jets, and no identified b-jets. Three different final states are considered based on the lepton flavour, namely $e^\pm e^\pm$, $e^\pm \mu ^\pm$, and $\mu ^\pm \mu ^\pm$. The lepton $p_{\text {T}}$  ($E_{\text {T}}^{\text {miss}}$) threshold is set to 30 (55) $\text {GeV}$ to reduce the SM background contributions, though the $E_{\text {T}}^{\text {miss}}$ criterion is not applied for the $\mu ^\pm \mu ^\pm$ final state due to the smaller $Z+$jets background expected in this channel. The leading (sub-leading) $p_{\text {T}}$ jet must have $p_{\text {T}} >30$ (20) $\text {GeV}$ and $|\eta |<2.5$. The two jets are required to have $65~\text {GeV}{}<m_{jj}<105$ $\text {GeV}$ and $|\Delta \eta _{jj}|<1.5$ in order to distinguish the signal from the $W^\pm W^\pm$ backgrounds, where $m_{jj}$ is the dijet invariant mass and $\Delta \eta _{jj}$ is the pseudorapidity separation between the two jets. The dilepton system is required to have $m_{\ell \ell }>40$ $\text {GeV}$ and in the case of the $e^\pm e^\pm$ final state, $m_{ee}$ must have $m_{ee}<80$ $\text {GeV}$ or $m_{ee}>100$ $\text {GeV}$ in order to suppress events with two opposite-sign prompt leptons where the charge of one of the electrons is misidentified. To reduce the contributions from $WZ+$jets and $ZZ+$jets production, events are removed if they contain additional leptons reconstructed with $p_{\text {T}} >6$ $\text {GeV}$ passing looser identification quality requirements, with a medium identification requirement for electrons as defined in Ref. [21] and the minimum identification required for muon reconstruction. Table 2 shows the kinematic selection criteria used for the $\ell \nu \ell \nu jj$ channel.

Table 2 Selection criteria for the $\ell \nu \ell \nu jj$ channel, split based on the lepton flavour: $e^\pm e^\pm$ , $e^\pm \mu ^\pm$ , and $\mu ^\pm \mu ^\pm$

4 Signal fiducial cross sections

At leading order (LO), the production of three W bosons can take place through radiation from a fermion, from an associated W and $Z/\gamma ^*/H$ production with the intermediate $Z/\gamma ^*/H$ boson decaying to two opposite-sign W bosons, or from a WWWW QGC vertex. Representative Feynman graphs for each of these production processes are shown in Fig. 1. Calculations are available including corrections at next-to-leading order (NLO) in QCD with all spin correlations involved in the vector-boson decays, the effects due to intermediate Higgs boson exchange, and off-shell contributions correctly taken into account [29]. Electroweak NLO corrections have been calculated recently [30]. However, they are not considered in this analysis.

In order to determine $W^{\pm }W^{\pm }W^{\mp }$ production cross sections, events are generated at NLO in QCD using MadGraph5_aMC@NLO  [31] including on-shell diagrams as well as Higgs associated diagrams. The CT10 NLO parton distribution function (PDF) [32] is used. Subsequent decays of unstable particles and parton showers are handled by pythia8 [33]. Fiducial cross sections are calculated using the generator-level lepton, jet, and $E_{\text {T}}^{\text {miss}}$ definitions as described in Ref. [34]. Generator-level prompt leptons (those not originating from hadron and $\tau$ lepton decays) are dressed with prompt photons within a cone of size $\Delta R = 0.1$. Generator-level jets are reconstructed by applying the anti-$k_t$ algorithm with radius parameter $R=0.4$ on all final-state particles after parton showering and hadronisation. The $E_{\text {T}}^{\text {miss}}$ variable is calculated using all generator-level neutrinos. The same kinematic selection criteria as listed in Tables 1 and 2 are applied on these objects, with the exception of the b-jet veto requirements in the $\ell \nu \ell \nu \ell \nu$ channel and the lepton quality requirements. To take into account the effect of the lepton isolation in the fiducial region, any lepton pairs must satisfy $\Delta R(\ell, \ell)>0.1$, and in the $\ell \nu \ell \nu jj$ channel any lepton-jet pairs must satisfy $\Delta R(j, \ell)>0.3$. Electrons or muons from $\tau$ decays are not included.

Fig. 1

Feynman graphs contributing at LO to $W^{\pm }W^{\pm }W^{\mp }$ production

The fiducial cross section is predicted to be $309 \pm 7~(\mathrm{stat.}) \pm {}15~(\mathrm{PDF}) \pm 8~(\mathrm{scale})$ ab in the $\ell \nu \ell \nu \ell \nu$ channel and $286 \pm 6~(\mathrm{stat.}) \pm {}15~(\mathrm{PDF}) \pm 10~(\mathrm{scale})$ ab in the $\ell \nu \ell \nu jj$ channel. Uncertainties due to the PDFs are computed using an envelope of the CT10, NNPDF3.0 [35], and MSTW2008 [36] NLO PDF 68 or 90% (for CT10) confidence level (CL) uncertainties, following the recommendation of Ref. [37]. The renormalization and factorization scales are set to the invariant mass of the WWW system. Scale uncertainties are estimated by varying the two scales independently up and down by a factor of two and taking the largest variation from the nominal cross-section values.

In order to combine the measurements from the two decay channels, a common phase space is defined where each W boson can decay either leptonically (including $\tau$ leptons) or hadronically, $pp\rightarrow W^\pm W^\pm W^\mp$ $+~X$, with no kinematic requirements placed on the final-state leptons but with jets restricted to have $p_{\text {T}} > 10$ $\text {GeV}$. The extrapolation factor from the fiducial phase space to the total phase space is large, but it is mainly due to the well-known W boson decay branching ratios. The total cross section in this common phase space is $241.5 \pm 0.1$ (stat.) $\pm 10.3$ (PDF) $\pm 6.3$ (scale) fb.

In order to determine the detector reconstruction effects on the signal selection, $W^{\pm }W^{\pm }W^{\mp }$ signal samples are generated with vbfnlo [29, 38,39,40] at LO. The parton shower and hadronisation are performed by pythia8. The fiducial cross sections are seen to be consistent between vbfnlo and MadGraph5_aMC@NLO when computed at the same order. The vbfnlo LO fiducial cross sections are normalized to the NLO fiducial cross section predicted by MadGraph5_aMC@NLO for the signal yield calculations. These events are processed through the full ATLAS detector simulation [41] based on Geant 4 [42]. To simulate the effect of multiple pp interactions occurring during the same or a neighbouring bunch crossing, minimum-bias interactions are generated and overlaid on the hard-scattering process. These events are then processed through the same object reconstruction and identification algorithms as used on data. MC events are reweighted so that the pile-up conditions in the simulation match the data. Additional corrections are made to the simulated samples to account for small differences between the simulation and the data for the object identification and reconstruction efficiencies, the trigger efficiencies, and the energy and momentum scales and resolutions. While excluded in the fiducial cross-section definition, the contribution from events with $W\rightarrow \tau \nu \rightarrow \ell \nu \nu \nu$ decays are counted as signal in the vbfnlo signal sample used in the final event selection. These events contribute up to 20% of the predicted signal yield. This approach is used to ease comparisons of the obtained cross-section limits with alternative cross-section predictions that may not simulate tau decays.

5 Backgrounds

5.1 Background estimation

The SM processes that mimic the $W^{\pm }W^{\pm }W^{\mp }$ signal signature can be grouped into five categories:

• The $WZ/\gamma ^*+$jets process that produces three prompt leptons or two prompt leptons with the same electric charge (referred to as “WZ background”);

• The $W\gamma +$jets or $Z\gamma +$jets processes where the photon is misreconstructed as a lepton (referred to as “$V\gamma$ background”, where $V=W, Z$);

• Processes other than $WZ/\gamma ^*+$jets that produce three prompt leptons or two prompt leptons with the same electric charge (referred to as “other prompt background”);

• Processes that produce two or three prompt charged leptons, but the charge of one lepton is misidentified (referred to as “charge-flip background”);

• Processes that have one or two non-prompt leptons originating either from misidentified jets or from hadronic decays (referred to as “fake-lepton background”).

The dominant irreducible background originates from the $WZ(\rightarrow \ell ^\pm \nu \ell ^\pm \ell ^\mp )+$jets process and is estimated using simulated events. In the $\ell \nu \ell \nu \ell \nu$ channel these events are generated with Powheg -BOX [43,44,45,46] and hadronised with pythia8 and in the $\ell \nu \ell \nu jj$ channel they are generated with Sherpa  [47]. In the $\ell \nu \ell \nu \ell \nu$ channel, the inclusive $WZ+$jets cross section is normalized using a scale factor ($1.08\pm 0.10$) derived from a WZ-enriched region in data. This region is obtained by requiring exactly one SFOS lepton pair with $|m_\mathrm{SFOS}-m_Z|<15~\text {GeV}$. In the $\ell \nu \ell \nu jj$ channel, the cross section is normalized to the NLO calculation in QCD from vbfnlo [48] in the specified fiducial phase space with a normalization factor of $1.04\pm 0.09$.

The $V\gamma$ background contributes when the photon is misidentified as an electron. In the $\ell \nu \ell \nu \ell \nu$ channel, this originates primarily from the $Z\gamma$ process and its contribution is estimated using events generated with Sherpa . In the $\ell \nu \ell \nu jj$ channel, this comes primarily from electroweak and strong production of $W\gamma jj$ events. Strong production of $W\gamma jj$ [49] is estimated using Alpgen  [50] interfaced to Herwig  [51] and Jimmy  [52] for simulation of the parton shower, fragmentation, hadronisation and the underlying event. The electroweak production of $W\gamma jj$ [53] is modelled using Sherpa .

Other SM processes that produce multiple prompt leptons include ZZ, $t\bar{t}V$, ZWW, ZZZ, $W^\pm W^\pm jj$ production, and double parton scattering processes. The production of ZZ is modelled with Powheg -BOX [46] and hadronised with pythia8 in the $\ell \nu \ell \nu \ell \nu$ channel and is modelled with Sherpa in the $\ell \nu \ell \nu jj$ channel. The $t\bar{t}V$ [54], ZWW [55], and ZZZ [55] processes are modelled using MadGraph5_aMC@NLO together with pythia8 for both channels. The non-resonant $W^\pm W^\pm jj$ background [56] is only important for the $\ell \nu \ell \nu jj$ channel and its contribution is estimated using Sherpa. Contributions from double parton scattering processes are found to be negligible in both channels.

The charge-flip background originates from processes where the charge of at least one prompt lepton is misidentified. This occurs primarily when a lepton from a hard bremsstrahlung photon conversion is recorded instead of the signal lepton. It mainly contributes to the 0-SFOS signal region in the $\ell \nu \ell \nu \ell \nu$ channel and the $e^\pm e^\pm$ / $e^\pm \mu ^\pm$ signal regions in the $\ell \nu \ell \nu jj$ channel. The electron charge misidentification rate is measured using $Z \rightarrow e^+e^-$ events. In the $\ell \nu \ell \nu \ell \nu$ channel, the charge-flip background is estimated by using these rates to re-weight the MC estimate of WZ and ZZ events based on the probability for opposite-sign events of this kind to migrate into the 0-SFOS category. In the $\ell \nu \ell \nu jj$ channel, the background is estimated by applying these rates on data events satisfying all signal selection criteria except the two leptons are required to have opposite-sign.

Contributions from fake-lepton backgrounds are estimated in data, using different approaches in the two channels. In the $\ell \nu \ell \nu \ell \nu$ channel, the probabilities of prompt leptons or non-prompt leptons to satisfy the signal lepton criteria are computed using a tag-and-probe method whereby a well-reconstructed “tag” lepton is used to identify the event and a second “probe” lepton is used to study the probabilities without bias. A tag lepton must satisfy the signal lepton requirements while a looser lepton selection criterion is defined for probe leptons with the lepton isolation requirements removed and the electron quality requirement loosened to medium as defined in Ref. [21]. The probability for a prompt lepton to satisfy the signal lepton criteria is estimated using $Z \rightarrow \ell ^+ \ell ^-$ events with the tag-and-probe lepton pair required to have the same-flavour, opposite-sign and an invariant mass within 10 $\text {GeV}$ of the pole mass of the Z boson. The probability for a non-prompt lepton from hadronic activity to satisfy the signal lepton requirement is estimated using the tag-and-probe method in a $W+$jets-enriched region with $E_{\text {T}}^{\text {miss}} > 10$ $\text {GeV}$, the tag lepton is a muon with $p_{\text {T}} > 40$ $\text {GeV}$, and the tag and probe leptons have the same electric charge. The probabilities are calculated separately for electrons and muons. A loosely identified set of data is also selected by requiring at least three loose leptons as defined above. This set of data, along with these probabilities are then used to estimate the background in the signal region with the matrix method [57].

In the $\ell \nu \ell \nu jj$ channel, events that contain one signal lepton and one “lepton-like” jet are selected. A “lepton-like” jet satisfies all signal lepton selection criteria except that the isolation requirements are $0.14<E_{\text {T}} ^{\mathrm {Cone}0.3}/p_{\text {T}} <2$ and $0.06<p_{\text {T}} ^{\mathrm {Cone}0.3}/p_{\text {T}} <2$ for electrons, and $0.07<E_{\text {T}} ^{\mathrm {Cone}0.3}/p_{\text {T}} <2$ and $0.07<p_{\text {T}} ^{\mathrm {Cone}0.3}/p_{\text {T}} <2$ for muons. In addition, the $|d_0/\sigma _{d_0}|$ and $|z_0 \times \sin \theta |$ selection criteria are loosened to 10 mm and 5 mm, respectively. These events are dominated by non-prompt leptons and are scaled by a fake factor to estimate the non-prompt background. The fake factor is the ratio of the number of jets satisfying the signal lepton identification criteria to the number of jets satisfying the “lepton-like” jet criteria. It is measured as a function of the jet $p_{\text {T}}$ and $\eta$ from a dijet-enriched sample selected by requiring a lepton back-to-back with a jet ($\Delta \phi _{j\ell } > 2.8$) and $E_{\text {T}}^{\text {miss}} < 40$ $\text {GeV}$.

5.2 Validation of background estimates

The background predictions are tested in several validation regions (VRs). These VRs are defined to be close to the signal region with a few selection criteria removed or inverted. They generally have dominant contributions from one or two background sources and a negligible contribution from the signal process. The signal and background predictions are compared to data for each VR in Table 3.

In the $\ell \nu \ell \nu \ell \nu$ channel, three VRs are considered. The first VR, called the pre-selection region, tests the modelling of the WZ+jets background by requiring exactly three signal leptons. The distribution of the trilepton transverse mass, $m_{T}^{3\ell } = \sqrt{2 p_{\text {T}} ^{3\ell }E_{\text {T}}^{\text {miss}} \Big (1-\cos (\phi ^{3\ell } - \phi ^{\vec {p}_{\text {T}}^{\text {~miss}}})\Big )}$ where $p_{\text {T}} ^{3\ell }$ is the $p_{\text {T}}$ of the trilepton system, is shown at the top left of Fig. 2. This VR includes the three signal regions (0, 1, and 2 SFOS), but the effect of the signal is considered negligible at this stage of the selection, as shown in Table 3. The WZ+jets purity is estimated to be around $70\%$ in this region. The second region, called the fake-lepton region, tests the modelling of the fake-lepton background by requiring exactly three signal leptons with no SFOS lepton pairs and at least one $b-$jet. The distribution of the jet multiplicity, $N_{\text {jet}}$, is shown at the top right of Fig. 2. The purity of the fake-lepton background is estimated to be around $80\%$ in this region. The third region, called $Z\gamma$ region, tests the modelling of the $Z\gamma$ background by requiring the presence of only $\mu ^+\mu ^- e^\pm$ events where the trilepton invariant mass is close to the Z resonance peak. This restricts the main contributions to originate from the $Z\gamma \rightarrow \mu ^+ \mu ^- \gamma$ and $Z \rightarrow \mu ^+ \mu ^- \rightarrow \mu ^+ \mu ^- \gamma$ processes. The $Z\gamma$ purity is estimated to be around $70\%$ in this region. The data are seen to be well described by the background in all three VRs.

In the $\ell \nu \ell \nu jj$ channel, five VRs are considered. The modelling of the charge-flip background is tested using $e^\pm e^\pm$ events with $80~\text {GeV}{}<m_{\ell \ell }<100~\text {GeV}$. The purity of the charge-flip background is estimated to be around $80\%$ in this region. The modelling of the $WZ+$jets background is checked in a $WZ+2$-jets region requiring the presence of an additional lepton. The $p_{\text {T}}$ of this third lepton is shown at the bottom left of Fig. 2. The purity of the WZ+jets is estimated to be around $60\%$ in this region. The modelling of backgrounds from non-prompt leptons is tested in a $b-$tagged region that requires at least one $b-$jet. The purity of the non-prompt lepton background is estimated to be around $80\%$ in this region. The $m_{jj}$ modelling is checked by examining events with masses $m_{jj}$ in the regions $m_{jj}<65~\text {GeV}$ or $m_{jj}>105~\text {GeV}$. The distribution of $m_{jj}$ in this region is shown at the bottom right of Fig. 2. Finally, conversion and prompt backgrounds are tested in a region with at most one jet, called the $\le 1$ jet region. The purity of the conversion and prompt backgrounds is estimated to be around $70\%$ in this region. As for the $\ell \nu \ell \nu \ell \nu$ channel, good agreement is observed between the data and the prediction in all five VRs.

Fig. 2

Distributions in four different VRs, two corresponding to the $\ell \nu \ell \nu \ell \nu$ channel (top) and two to the $\ell \nu \ell \nu jj$ channel (bottom). For the $\ell \nu \ell \nu \ell \nu$ channel the $m_T^{3\ell }$ distribution in the preselection region (top left) and the jet multiplicity distribution in the fake-lepton region (top right) are shown. For the $\ell \nu \ell \nu jj$ channel the third-lepton $p_{\text {T}}$ in the $WZ+2$-jets region (bottom left) and the $m_{jj}$ distribution in the W mass sideband region (bottom right) are shown. The “other backgrounds” contain prompt leptons and are estimated from MC. The hashed band represents total uncertainties on the signal-plus-background prediction. The highest bin also includes events falling out of the range shown

Table 3 Expected numbers of signal and background events in the VRs compared to the numbers of events observed in data. The first uncertainty is statistical and the second is systematic

6 Systematic uncertainties

Systematic uncertainties in the signal and background predictions arise from the measurement of the integrated luminosity, from the experimental and theoretical modelling of the signal acceptance and detection efficiency, and from the background estimation. The effect of the systematic uncertainties on the overall signal and background yields are evaluated separately for the $\ell \nu \ell \nu \ell \nu$ and $\ell \nu \ell \nu jj$ channels . The results are summarised in Table 4. The systematic uncertainties are included as nuisance parameters in the profile likelihood described in Sect. 7. Correlations of systematic uncertainties arising from common sources are maintained across signal and background processes and channels.

Table 4 The effect of the various systematic uncertainties on the total signal and background yields (in percent) for both channels

The experimental uncertainties include the uncertainties on the lepton and jet energy and momentum scales and resolutions, on the efficiencies of the lepton and jet reconstruction and identification, and on the modelling of $E_{\text {T}}^{\text {miss}}$ and b-jets. They are evaluated separately for both the signal and background estimations. For the expected signal yield, the major contributions in the $\ell \nu \ell \nu \ell \nu$ channel come from uncertainties in the lepton reconstruction and identification efficiencies as well as lepton energy/momentum resolution and scale modelling ($\pm 1.6\%$), $E_{\text {T}}^{\text {miss}}$ modelling ($\pm 1.1\%$), and jet energy scale and resolution ($\pm 2.3\%$). The contributions in the $\ell \nu \ell \nu jj$ channel come from uncertainties in the lepton efficiencies and energy/momentum modelling ($\pm 2.1\%$), $E_{\text {T}}^{\text {miss}}$ modelling ($\pm 0.7\%$), b-jet identification ($\pm 2.2\%$), and jet energy resolution and scale modelling ($\pm 21\%$). Larger systematic uncertainties due to the jet energy scale and resolution are expected in the $\ell \nu \ell \nu jj$ channel due to the dijet requirements, in particular in the dijet invariant mass. For the background yields estimated from MC simulation, the major contributions in the $\ell \nu \ell \nu \ell \nu$ channel come from uncertainties in lepton reconstruction and identification efficiencies ($\pm 1.8\%$), $E_{\text {T}}^{\text {miss}}$ modelling ($\pm 1.4\%$), and jet energy modelling ($\pm 2.8\%$). The major contributions in the $\ell \nu \ell \nu jj$ channel come from uncertainties in lepton efficiencies and energy modelling ($\pm 3.3\%$), $E_{\text {T}}^{\text {miss}}$ modelling ($\pm 1.8\%$), b-jet identification ($\pm 2.2\%$), and jet energy modelling ($\pm 15\%$).

The estimates of the data-driven fake-lepton background also have uncertainties specific to each channel. In the $\ell \nu \ell \nu \ell \nu$ channel, the systematic uncertainty results from the uncertainties on the probabilities of candidate leptons that satisfy the looser lepton selection criteria to also satisfy the signal lepton selection criteria. For prompt leptons this uncertainty is ±(5 to 10)% while for fake leptons and misidentified/non-prompt leptons this uncertainty is ±(80 to 90)%. The latter uncertainty is a conservative estimate which accounts for differences in the heavy-flavour and light-flavour composition between the signal region and the control region where the fake-lepton efficiency is determined for these leptons. In the $\ell \nu \ell \nu jj$ channel, the systematic uncertainty results from the uncertainties in the measurement of the fake factors, which is estimated to be ±(20 to 30)%. Statistical uncertainties in the samples used for the matrix method and the fake-factor method also contribute to the overall uncertainty of the estimation of the fake-lepton background. The total uncertainty in the overall fake-lepton background yield is $\pm 13\%$ in the $\ell \nu \ell \nu \ell \nu$ channel and $\pm 8\%$ in the $\ell \nu \ell \nu jj$ channel.

The charge-flip background is only relevant for the $e^\pm e^\pm$/$e^\pm \mu ^\pm$ final state in the $\ell \nu \ell \nu jj$ channel and for the 0-SFOS region in the $\ell \nu \ell \nu \ell \nu$ channel. Its uncertainty is dominated by the statistical precision with which the electron charge misidentification rate is determined from the available data. Since the charge-flip background estimation uses the number of $Z(\rightarrow e^+ e^-)+2$ jets events, the number of events in the data also contributes to the overall systematic uncertainty. In the $\ell \nu \ell \nu \ell \nu$ channel, the uncertainty on the charge-flip background estimate is $\pm 0.5\%$ in the 0-SFOS region but is $\pm 0.04\%$ for the total background estimate in all three signal regions. In the $\ell \nu \ell \nu jj$ channel, the total systematic uncertainty in the overall background yield due to the uncertainty in the charge-flip background estimate is found to be $\pm 2.2\%$.

There are also uncertainties in the overall normalization of the signal and MC background cross sections. Uncertainties in the signal cross section are those described in Sect. 4. These are not, however, included as uncertainties in the model and merely serve as a comparison for the final measurement in Sect. 7. The normalizations of the SM background cross sections described in Sect. 5.1 have their own associated uncertainties. The uncertainty in the predicted WZ+jets background cross section is the most important one since it is the largest irreducible background. The size of the uncertainty relative to the predicted WZ+jets background is $\pm 10\%$ in the $\ell \nu \ell \nu \ell \nu$ channel and ±(16 to 23)% in the $\ell \nu \ell \nu jj$ channel depending on the production mechanism. This uncertainty is based on the measurement performed in the control region, for the $\ell \nu \ell \nu \ell \nu$ channel as described in Sect. 5, while it is a combination of the scale, PDF and parton shower uncertainties estimated as in Ref. [12], for the $\ell \nu \ell \nu jj$ channel. The remaining uncertainties are mostly negligible in the overall background prediction. The normalization uncertainty in the total background prediction is around $\pm 8$% in the $\ell \nu \ell \nu \ell \nu$ channel and $\pm 13$% in the $\ell \nu \ell \nu jj$ channel.

The uncertainty on the integrated luminosity is $\pm 1.9\%$, affecting the overall normalization of both the signal and background processes estimated from MC simulation. It is derived following the methodology detailed in Ref. [18]. The uncertainties associated with the pile-up reweighting of the events are estimated to be no more than $\pm 0.1\%$ for the signal and the backgrounds.

7 Cross-section measurement

The signal and background predictions together with their uncertainties are compared to the data for six signal regions in Table 5. The expected signal yields are calculated using the SM $W^{\pm }W^{\pm }W^{\mp }$ cross sections listed in Sect. 4. The expected numbers of signal plus background events are consistent with the numbers of events observed in data in all regions. Figure 3 shows the $m_{T}^{3\ell }$ distribution for the $\ell \nu \ell \nu \ell \nu$ channel and the distribution of the sum of the scalar $p_{\text {T}}$ for all selected objects, $\Sigma p_{\text {T}} = p_{\text {T}} ^{\ell ,1}+p_{\text {T}} ^{\ell ,2}+p_{\text {T}} ^{j,1}+p_{\text {T}} ^{j,2}+E_{\text {T}}^{\text {miss}}$, for the $\ell \nu \ell \nu jj$ channel, after summing over the three signal regions in each channel. Good agreement between data and the signal-plus-background model is observed for both distributions.

Table 5 Numbers of expected signal and background events, and their statistical and systematic uncertainties, together with the observed yields in the data in the signal regions for the two channels

The amount of $W^{\pm }W^{\pm }W^{\mp }$ signal in the selected data set is determined using the numbers of expected signal and background events as well the numbers of observed events in the data. The signal strength, $\mu$, is the parameter of interest, defined as a scale factor multiplying the cross section times branching ratio predicted by the SM. A test statistic based on the profile-likelihood ratio [58] is used to extract $\mu$ from a maximum-likelihood fit of the signal-plus-background model to the data. The likelihood, $\mathcal {L}$, is given by

$$\begin{aligned} \mathcal{{L}} = \prod _c \prod _i {\text {Poisson}} \left[ n_{i_c}^{\text {obs}} \mid \mu \times n_{i_c}^{\text {sig, SM}} (\theta _k) + n_{i_c}^{\text {bkg}}(\theta _k) \right] \prod _k g(\theta _k) \end{aligned}$$

(1)

where the index c represents one of the two analysis channels, i represents one of the three signal regions in each channel, $n^{\text {obs}}$ is number of observed events, $n^{\text {sig, SM}}$ is the expected number of signal events based on the SM calculations, and $n^{\text {bkg}}$ is the expected number of background events. The effect of a systematic uncertainty k on the likelihood is modelled with a nuisance parameter, $\theta _k$, constrained with a corresponding Gaussian probability density function $g(\theta _k)$.

The test statistic, $t_\mu$, is defined as

$$\begin{aligned} t_{\mu } = -2 \ln \lambda (\mu )=-2 \ln \frac{\mathcal{{L}}(\mu , \hat{\hat{\theta }}(\mu ))}{\mathcal{{L}}(\hat{\mu }, \hat{\theta })} \end{aligned}$$

(2)

where $\hat{\mu }$ is the unconditional maximum-likelihood (ML) estimators of the independent signal strength in the categories being compared, $\hat{\theta }$ are the unconditional ML estimators of the nuisance parameters, and $\hat{\hat{\theta }}(\mu )$ are the conditional ML estimators of $\theta$ for a given value of $\mu$. The significance of $\mu$ is obtained with the above test statistic, and is estimated using 100,000 MC pseudo-experiments to determine how well the fit result agrees with the background-only hypothesis. The observed (expected) significance of a positive signal cross section is $0.96~\sigma$ ($1.05~\sigma$) for the combination of the two channels. Most of the sensitivity comes from the 0-SFOS category in the $\ell \nu \ell \nu \ell \nu$ channel and the $\mu ^\pm \mu ^\pm$ category in the $\ell \nu \ell \nu jj$ channel. The most significant deviation from the signal-plus-background hypothesis occurs in the $e^\pm e^\pm$ region, where zero events are observed and 4.0 background and 0.46 signal events are expected. The probability that the background fluctuates down to zero events is 2.3%.

The central value of $\mu$ corresponds to the minimum of the negative log-likelihood distribution. The measured fiducial cross section in each channel is obtained by multiplying $\mu$ by the expected value of the fiducial cross section in that channel. The measured total cross section is obtained by combining the results for the two channels and then extrapolating to the total phase space using the signal acceptance. The log-likelihood scans for the total cross-section measurement are evaluated with and without systematic uncertainties and are shown in Fig. 4. The expected and observed fiducial and total cross sections are summarized in Table 6.

The presence of the $W^{\pm }W^{\pm }W^{\mp }$ signal is also assessed using a one-sided 95% CL upper limit on the production cross section using the CL$_s$ method of Ref. [59]. The limits are evaluated using 2000 MC pseudo-experiments. The observed (expected) upper limit on the fiducial cross section in the absence of $W^{\pm }W^{\pm }W^{\mp }$ production is found to be 1.3 fb (1.1 fb) in the $\ell \nu \ell \nu \ell \nu$ channel and 1.1 fb (0.9 fb) in the $\ell \nu \ell \nu jj$ channel. The observed (expected) upper limit in the absence of $W^{\pm }W^{\pm }W^{\mp }$ production on the total cross section is 730 fb (560 fb) when the two channels are combined. If the SM $W^{\pm }W^{\pm }W^{\mp }$ signal is also considered, the expected upper limit on the total cross section is 850 fb.

Table 6 The predicted and observed fiducial cross sections for the $\ell \nu \ell \nu \ell \nu$ and $\ell \nu \ell \nu jj$ channels and the predicted and observed total cross section for the combination of the two channels

Fig. 3

The distribution of $m_T^{3\ell }$ for the $\ell \nu \ell \nu \ell \nu$ channel (left) and the distribution of $\Sigma p_{\text {T}}$ for the $\ell \nu \ell \nu jj$ channel (right) as observed in the data (dots with error bars indicating the statistical uncertainties) and as expected from SM signal and background processes. The ratios between the observed numbers of events in data and the expected SM signal plus background contributions are shown in the lower panels. The hashed bands results from the systematic uncertainties on the sum of the signal plus background contributions. The “other backgrounds” contain prompt leptons and are estimated from MC. Contributions from aQGCs are also shown, assuming the non-unitarized case ($\Lambda _\mathrm{FF} = \infty$) and two different sets of $f_{S,0}/\Lambda ^4$ and $f_{S,1}/\Lambda ^4$ configurations ($f_{S,0}/\Lambda ^4{}=2000~\text {TeV}{}^{-4}$, $f_{S,1}/\Lambda ^4{}=2000~\text {TeV}{}^{-4}$ and $f_{S,0}/\Lambda ^4{}=2000~\text {TeV}{}^{-4}$, $f_{S,1}/\Lambda ^4{}=-6000~\text {TeV}{}^{-4}$). The highest bin also includes events falling out of the range shown

Fig. 4

Profile-likelihood scans as a function of the total cross section for the combination of all six signal regions. The expected (red) scans are shown when considering only statistical uncertainties (dashed line) and when considering both statistical and systematic uncertainties (solid line). The observed (black solid line) scan is also shown. The dotted black grid-lines pinpoint the location of the 68 and 95% CL uncertainties in the measurement of the signal strength

8 Limits on anomalous quartic gauge couplings (aQGCs)

Contributions from sources beyond the SM to the $W^{\pm }W^{\pm }W^{\mp }$ production process can be expressed in a model-independent way using higher-dimensional operators leading to WWWW aQGCs. The parameterization of aQGCs is based on Ref. [60] in a linear representation [61] considering only dimension-eight operators involving four gauge bosons. There are 18 dimension-eight operators built from the covariant derivative of the Higgs field $D_\mu \Phi$, the SU(2)$_{\text {L}}$ field strength $W^i_{\mu \nu }$, and U(1)$_{\text {Y}}$ field strength $B_{\mu \nu }$. Only the two terms built exclusively from $D_\mu \Phi$ and with aQGC parameters $f_{S,0}/\Lambda ^4$ and $f_{S,1}/\Lambda ^4$ are considered in this analysis:

$$\begin{aligned} \mathcal{{L}}_{S,0} = \frac{f_{S,0}}{\Lambda ^4} [(D_\mu \Phi )^\dagger D_\nu \Phi ] \times [(D^\mu \Phi )^\dagger D^\nu \Phi ], \end{aligned}$$

(3)

$$\begin{aligned} \mathcal{{L}}_{S,1} = \frac{f_{S,1}}{\Lambda ^4} [(D_\mu \Phi )^\dagger D^\mu \Phi ] \times [(D_\nu \Phi )^\dagger D^\nu \Phi ], \end{aligned}$$

(4)

where $\Lambda$ is the energy scale of the new physics. These two operators only affect massive bosons and do not depend on the gauge boson momenta since no SU(2)$_{\text {L}}$ or U(1)$_{\text {Y}}$ field strengths are included. As a result, they are important for the study of longitudinal vector-boson scattering. Similar parameters were studied before by the ATLAS and CMS Collaborations in Refs. [8, 10, 16]

The effective Lagrangian approach leads to tree-level unitarity violation. This can be avoided by introducing a form factor [62] as

$$\begin{aligned} \alpha \rightarrow \frac{\alpha _0}{(1+\hat{s}/\Lambda _\mathrm{FF}^2)} \end{aligned}$$

(5)

where $\alpha$ corresponds to one of the two couplings, $\alpha _0$ is the value of the aQGC at low energy, $\hat{s}$ is the square of the partonic centre-of-mass energy, and $\Lambda _\mathrm{FF}$ is the form-factor cutoff scale. However, there is no theoretical algorithm to predict for which form-factor cutoff scale the cross section would violate unitarity. Therefore different values of $\Lambda _\mathrm{FF}$ are considered with $\Lambda _\mathrm{FF}=$ 0.5, 1, 2, and 3 $\text {TeV}$ as well as $\Lambda _\mathrm{FF} = \infty$, which corresponds to the non-unitarized case.

Events with aQGCs are generated with vbfnlo at LO and passed through the ATLAS detector simulation. A grid of samples is obtained using different parameters of $f_{S,0}/\Lambda ^4{}$ and $f_{S,1}/\Lambda ^4{}$ values. The interpolation between these points is performed with a 2-dimensional quadratic function in the ($f_{S,0}/\Lambda ^4{}$, $f_{S,1}/\Lambda ^4{}$) space. The LO samples are scaled using a factor derived from the ratio of the SM LO to NLO predictions. Figure 3 show the expected distribution for the non-unitarized ($\Lambda _\mathrm{FF} = \infty$) aQGC signal samples being generated with parameters $f_{S,0}/\Lambda ^4{}=2000~\text {TeV}{}^{-4}$, $f_{S,1}/\Lambda ^4{}=2000~\text {TeV}{}^{-4}$ in red and parameters $f_{S,0}/\Lambda ^4{}=2000~\text {TeV}{}^{-4}$, $f_{S,1}/\Lambda ^4{}=-6000~\text {TeV}{}^{-4}$ in blue as a function of the $m_T^{3\ell }$ distribution in the $\ell \nu \ell \nu \ell \nu$ channel and the $\Sigma p_{\text {T}}$ distribution in the $\ell \nu \ell \nu jj$ channel, summed over the three signal regions in each channel. Even though aQGC events tend to have leptons or jets with larger momenta, the detection efficiency for events in the fiducial region is found to be consistent with the one obtained for the SM sample within 20%. The efficiencies of the aQGC samples are used with their statistical and systematic uncertainties to derive the 95% confidence intervals (CI) on aQGC, while the largest observed deviation of the aQGC efficiencies from the SM one is used as an extra systematic uncertainty. Frequentist CI on the anomalous coupling are computed by forming a profile-likelihood-ratio test that incorporates the observed and expected numbers of signal events for different values of the anomalous couplings. Table 7 shows the expected and observed 95% CI on $f_{S,0}/\Lambda ^4$ ($f_{S,1}/\Lambda ^4$) with different $\Lambda _\mathrm{FF}$ values, assuming $f_{S,1}/\Lambda ^4$ ($f_{S,0}/\Lambda ^4$) to be zero. Figure 5 shows the two-dimensional 95% CL contour limits of $f_{S,0}/\Lambda ^4$ vs $f_{S,1}/\Lambda ^4$ in the cases where $\Lambda _\mathrm{FF} = 1$ $\text {TeV}$ and $\Lambda _\mathrm{FF} = \infty$. For $\Lambda _\mathrm{FF} = \infty$, the limits can be compared to the stronger limits obtained by the CMS Collaboration in Ref. [16] in a different production channel. Other parameterization ($\alpha _4$, $\alpha _5$) of new physics have been introduced in Refs. [63,64,65]. The limits presented in this paper can be converted into limits on $\alpha _4$ and $\alpha _5$ following the formalism defined in the Appendix of Ref. [60] and using Equations (60) and (61) in Ref. [66]. For example, non-unitarized limits obtained for $\Lambda _\mathrm{FF} = \infty$ are: $\alpha _4$ expected [$-0.61$, 0.78], $\alpha _4$ observed [$-0.49$, 0.75] and $\alpha _5$ expected [$-0.57$,0.69], $\alpha _5$ observed [$-0.48$,0.62]. Limits derived by the ATLAS Collaboration in other final states are reported in Refs. [8, 10]. The latter were obtained using a different unitarization scheme. Since that scheme is not applicable to triboson production, a combination of the limits is not possible.

Table 7 Expected and observed 95% CI on $f_{S,0}/\Lambda ^4$ ($f_{S,1}/\Lambda ^4$) for different $\Lambda _\mathrm{FF}$ values, assuming $f_{S,1}/\Lambda ^4$ ($f_{S,0}/\Lambda ^4$) to be zero

Fig. 5

Expected 68 and 95% CL contours for $f_{S,1}/\Lambda ^4$ vs $f_{S,0}/\Lambda ^4$ compared to the observed 95% CL contour and the observed best-fit value for cases when $\Lambda _\mathrm{FF}=1$ $\text {TeV}$ (left) and $\Lambda _\mathrm{FF}=\infty$ (right)

9 Summary

A search for triboson $W^{\pm }W^{\pm }W^{\mp }$ production in two decay channels (${W^{\pm }W^{\pm }W^{\mp } \rightarrow \ell ^\pm \nu \ell ^\pm \nu \ell ^\mp \nu }$ and ${W^{\pm }W^{\pm }W^{\mp } \rightarrow \ell ^\pm \nu \ell ^\pm \nu jj}$ with $\ell =e, \mu$) is reported, using proton-proton collision data corresponding to an integrated luminosity of 20.3 $\mathrm{fb}^\mathrm{-1}$ at a centre-of-mass energy of 8 $\text {TeV}$ collected by the ATLAS detector at the LHC. Events with exactly three charged leptons or two same-charge leptons in association with two jets are selected. The data are found to be in good agreement with the SM predictions in all signal regions. The observed 95% CL upper limit on the SM $W^{\pm }W^{\pm }W^{\mp }$ production cross section is found to be 730 fb with an expected limit of 560 fb in the absence of $W^{\pm }W^{\pm }W^{\mp }$ production. Limits are also set on the aQGC parameters $f_{S,0}/\Lambda ^4$ and $f_{S,1}/\Lambda ^4$.