1 Introduction

The \(\overline{\partial }\)-complex plays an important role in the theory of several complex variables since many important results for holomorphic functions can be obtained by solving nonhomogeneous \(\overline{\partial }\)-equation. We obtain \(\overline{\partial }_b\)-complex when it is restricted to a CR submanifold, and many important results for CR functions can also be obtained by solving \(\overline{\partial }_b\)-equation. In general, for a differential complex, there is an abstract way to obtain a boundary complex restricted to a submanifold, which is written down in terms of quotient sheafs (cf., e.g., [3, 4, 25]).

In quaternionic analysis, we now know the k-Cauchy–Fueter complexes explicitly (cf. [1, 5, 8, 9, 11, 30, 35, 41] and references therein), which are used to show several interesting properties of k-regular functions (cf. [12, 35, 40, 42] and references therein). When restricted to a quadratic hypersurface in \(\mathbb {H}^{n+1},\) we have the tangential k-Cauchy–Fueter operators and k-CF functions (cf. [39] for \(k=1,n=2\)), corresponding to \(\overline{\partial }_b\) and CR functions over a CR manifold. In this paper, we will consider their restriction to a model quadratic hypersurface

$$\begin{aligned} \mathcal {S}:=\{(q',q_{n+1})\in \mathbb {H}^{n}\times \mathbb {H}:\rho (q',q_{n+1})=0\} \end{aligned}$$
(1.1)

in \(\mathbb {H}^{n+1},\) where

$$\begin{aligned} \rho (q',q_{n+1}):=\mathrm{Re}\,q_{n+1}-\phi (q'),\qquad \phi (q'):=\sum _{l=0}^{n-1}\left( -3x^2_{4l+1}+x^2_{4l+2} +x^2_{4l+3}+x^2_{4l+4}\right) .\nonumber \\ \end{aligned}$$
(1.2)

Here, we write \(q'=(\ldots ,q_l,\ldots ),q_l=x_{4l+1}+\mathbf {i}x_{4l+2}+ \mathbf {j}x_{4l+3}+\mathbf {k}x_{4l+4}.\) This hypersurface has the structure of the right quaternionic Heisenberg group\(\mathscr {H}=\mathbb {H}^n\times {\mathrm{Im}}\ \mathbb {H}\) with the multiplication given by

$$\begin{aligned} (x,t)\cdot (y,s)=\left( x+y,t+s+2{{\mathrm{Im}}} ({x}\overline{y})\right) , \end{aligned}$$
(1.3)

where \(x,y\in \mathbb {H}^{n}\) and \({t},s\in {\mathrm{Im}}\ \mathbb {H}.\) We construct a family of differential complexes on \( \mathscr {H} \), the tangential k-Cauchy–Fueter complexes, given by

$$\begin{aligned} \begin{aligned} 0{\rightarrow } C^{\infty }(\Omega ,\mathscr {V}_0)&\xrightarrow {\mathscr {D}_{0}} C^{\infty }(\Omega ,\mathscr {V}_1)\xrightarrow {\mathscr {D}_{1}} C^{\infty }(\Omega ,\mathscr {V}_2)\rightarrow \cdots \xrightarrow {\mathscr {D}_{2n-2}} C^{\infty }(\Omega ,\mathscr {V}_{2n-1})\rightarrow 0, \end{aligned}\nonumber \\ \end{aligned}$$
(1.4)

for a domain \(\Omega \) in \(\mathscr {H} \), where

$$\begin{aligned} \begin{aligned} \mathscr {V}_j:=&\odot ^{k-j}\mathbb {C}^{2}\otimes \wedge ^j\mathbb {C}^{2n},\qquad j=0,1,\ldots , k, \\ \mathscr {V}_{j}:=&\odot ^{j-k-1}\mathbb {C}^{2}\otimes \wedge ^{j+1}\mathbb {C}^{2n}, \qquad j=k+1,\ldots ,2n-1, \end{aligned} \end{aligned}$$
(1.5)

for fixed \(k=0,1,\ldots \), and \(\odot ^{p}\mathbb {C}^{2}\) is the pth symmetric power of \(\mathbb {C}^2.\) They are the quaternionic counterpart of \(\bar{\partial }_b\)-complex over the Heisenberg group in the theory of several complex variables. They have the same form as the k-Cauchy–Fueter complexes on \(\mathbb {H}^n\) (cf. Remark 2.1), but \(\mathscr {D}_{j}\)’s are given in terms of left invariant vector fields (2.23) (2.26) (2.27), which are differential operators of variable coefficients. So we cannot use the computational algebraic method in [12] to construct these complexes. This family of complexes are natural in the sense that they can be viewed as the restriction to the hypersurface \(\mathcal {S}\) of complexes on \(\mathbb {H}^{n+1},\) but not natural in the sense that they are not invariant under the conformal transformation group \(\mathrm{Sp}( n+1,1)\) of \({\mathscr {H}}\) (cf. Sect. 2.5).

\(\mathscr {D}_{0}\) in (1.4) is called the tangential k-Cauchy–Fueter operator. A \(\odot ^{k}\mathbb {C}^{2}\)-valued distribution f on \(\Omega \) is called k-CF if \(\mathscr {D}_{0}f=0\) in the sense of distributions. The space of all k-CF functions on \(\Omega \) is denoted by \(\mathcal {A}_k(\Omega ).\) A 1-CF function is also called anti-CRF function in [18, 19]. Such functions play an important role in the study of pseudo-Einstein equation over the quaternionic Heisenberg group [19].

On the other hand, when the hypersurface is the boundary of the Siegel upper half space, i.e., the defining function in (1.1) is given by

$$\begin{aligned} \rho =\mathrm{Re}\, q_{n+1}-|q'|^2, \end{aligned}$$

the corresponding group is the left quaternionic Heisenberg group\(\widetilde{{\mathscr {H}}}:=\mathbb {H}^n\times \mathrm{Im}\,\mathbb {H}\) with the multiplication given by

$$\begin{aligned} (x,t)\cdot (y,s)=\left( x+y,t+s+2{{\mathrm{Im}}} (\overline{x}{y})\right) . \end{aligned}$$
(1.6)

We already know the tangential k-Cauchy–Fueter complex (cf. [37, Theorem 1.0.1]) on the left quaternionic Heisenberg group by using the twistor method (see also [6, 27] for constructing complexes by this method) . But in this case, \(\wedge ^j\mathbb {C}^{2n}\) in (1.5) must be replaced by the irreducible representation of \(\mathfrak {sp}(2n,\mathbb {C})\) with the highest weight to be the jth fundamental weight (cf. Sect. 2.5). Since it is more complicated than the right case, we only consider the right quaternionic Heisenberg group in this paper. We see that when restricted to different submanifolds, we get different differential complexes. This is a new phenomenon compared to several complex variables, where expressions of \(\overline{\partial }_b\)-complex for different CR submanifolds are the same. It is an interesting problem to write down explicitly the tangential k-Cauchy–Fueter complexes for all quadratic hypersurfaces in \(\mathbb {H}^{n+1}\) (cf. [39] for such hypersurfaces).

In this paper, we prove Hartogs’ phenomenon for k-CF functions over right quaternionic Heisenberg group.

Theorem 1.1

Let \(\Omega \) be a bounded open set in the right quaternionic Heisenberg group \(\mathscr {H}\) with \(\mathrm{dim}\ \mathscr {H}\ge 19,\) and let K be a compact subset of \(\Omega \) such that \(\Omega {\setminus } K\) is connected. Then, for each \(u\in \mathcal {A}_k(\Omega {\setminus } K),\)\(k=2,3,\ldots ,\) we can find \(U\in \mathcal {A}_k(\Omega )\) such that \(U=u\) in \(\Omega {\setminus } K.\)

The restriction of \(\mathrm{dim}\ \mathscr {H} \) and k in this theorem comes from the technical difficulty to establish the \(L^2\) estimate in the remaining cases. A form of Hartogs’ phenomenon was proved for many elliptic differential systems (cf. [12, 26] and references therein). Notably, in our case \(\mathscr {D}_0\) as a matrix-valued horizontal vector field is not an elliptic system, and (1.4) is not an elliptic complex. This is because symbols of \(\mathscr {D}_j\)’s vanish at the cotangent vectors annihilating horizontal vector fields.

In the complex case, we have deep Hartogs–Bochner effect for CR functions on CR submanifolds, which are usually proved by using integral representation formulae (cf. [15, 23, 29] and references therein for further development of this effect). But in the quaternionic case, the integral representation formulae are not sufficiently developed, and only Bochner–Martinelli-type formulae are known (cf. [34, 35]). As in the theory of several complex variables, the formulae with Bochner–Martinelli- type kernels are not good enough to prove the extension phenomenon.

Given a differential complex, it is a fundamental problem to investigate its cohomology group or its Poincaré lemma over a domain (cf., e.g., [7, 16]). In particular, we hope to solve the nonhomogeneous tangential k-Cauchy–Fueter equation

$$\begin{aligned} \mathscr {D}_0u=f, \end{aligned}$$
(1.7)

for \(f\in L^2(\mathscr {H},\mathscr {V}_1)\), under the compatibility condition

$$\begin{aligned} \mathscr {D}_1f=0, \end{aligned}$$
(1.8)

i.e., f is \(\mathscr {D}_1\)-closed. If we can find compactly supported solution of (1.7)–(1.8) when f is compactly supported, it is a standard procedure to derive Hartogs’ phenomenon (cf., e.g., [17, 35]). One way to solve (1.7)–(1.8) is to consider the associated Hodge–Laplacian

$$\begin{aligned} \begin{aligned} \Box _1 =\mathscr {D}_0\mathscr {D}_0^*+\mathscr {D}_1^*\mathscr {D}_1:L^2(\mathscr {H},\mathscr {V}_1)\rightarrow L^2(\mathscr {H}, \mathscr {V}_1). \end{aligned} \end{aligned}$$
(1.9)

By identifying \(\mathscr {V}_1=\odot ^{k-1}\mathbb {C}^2\otimes \mathbb {C}^{2n}\) with \(\mathbb {C}^{2nk},\) we can see that \(\Box _1\) is a \((2kn)\times (2kn)\) matrix-valued differential operator of the second order, which is not diagonal (cf. Appendix for the expression in the case \(n=2,k=2\)). So it is not easy to verify the subellipticity of \(\Box _1\) and find its fundamental solution, while in the complex case, the Hodge–Laplacian associated with \(\overline{\partial }_b\)-complex is diagonal and it is easy to find its fundamental solution (cf. [13]).

By using the \(L^2\) method, we establish the following estimate: when \(\mathrm{dim}\ \mathscr {H}\ge 19,\) there exists some constant \(c>0\) such that

$$\begin{aligned} \Vert \mathscr {D}_{0}^*f\Vert ^2+\Vert \mathscr {D}_{1}f\Vert ^2\ge c\langle \Delta _bf,f\rangle \end{aligned}$$
(1.10)

for \(f\in C^2\left( \mathscr {H},\mathscr {V}_1\right) \cap L^2\left( \mathscr {H},\mathscr {V}_1\right) ,\) where \(\Delta _b\) is the SubLaplacian on the right quaternionic Heisenberg group. But \(\langle \Delta _bf,f\rangle \) does not control the \(L^2\) norm of f. It only controls \(\Vert f\Vert ^2_{L^{\frac{Q+2}{Q-2}}}\) by the well-known Sobolev inequality [19], where \(Q=4n+6\) is the homogeneous dimension of \(\mathscr {H}.\) To avoid this difficulty, we consider the locally flat compact manifold \(\mathscr {H}/\mathscr {H}_{\mathbb {Z}},\) where

$$\begin{aligned} \mathscr {H}_{\mathbb {Z}}:=\mathbb {Z}^{4n+3} \end{aligned}$$
(1.11)

is a lattice of \(\mathscr {H}.\) It is a spherical qc manifold (cf. [31]). Because the self-adjoint subelliptic operator \(\Delta _b\) over a compact manifold has discrete spectra, \(\langle \Delta _bf,f\rangle \) controls the \(L^2\) norm of f for \(f\perp \mathrm{ker}\,\Delta _b.\) Moreover, by the Poincaré-type inequality we can show \(\mathrm{ker}\,\Delta _b\) consisting of constant vectors. Namely, there exists some \(c''>0\) such that

$$\begin{aligned} \langle \Delta _bf,f\rangle \ge c''\Vert f\Vert ^2 \end{aligned}$$
(1.12)

for \(f\in C^2(\mathscr {H}/\mathscr {H}_{\mathbb {Z}},\mathscr {V}_1)\) and \(f\perp \) constant vectors. It is a standard way to use the \(L^2\) estimate to solve the nonhomogeneous tangential k-Cauchy–Fueter equation (1.7)–(1.8) on \(\mathscr {H}/\mathscr {H}_\mathbb {Z}\). The solution has an important vanishing property which allows us to prove Hartogs’ phenomenon. See also [13] for the existence theorem for \(\overline{\partial }_b\)-equation over compact CR manifolds by establishing a priori estimate.

In Sect. 2, we give preliminaries on the right quaternionic Heisenberg group, the horizontal complex vector fields \(Z_A^{A'}\)’s and nice behavior of their commutators. We also give the definition of the tangential k-Cauchy–Fueter operators and their basic properties. It is checked directly that (1.4)–(1.5) is a complex. We compare the complexes on the left and right quaternionic Heisenberg groups. In Sect. 3, we use integration by part and Poincaré-type inequality to show the \(L^2\) estimate (1.10) (1.12) for the tangential k-Cauchy–Fueter operator. In Sect. 4, we use the \(L^2\) estimate to solve the nonhomogeneous tangential k-Cauchy–Fueter equation (1.7)–(1.8) over the quotient manifold \(\mathscr {H}/\mathscr {H}_{\mathbb {Z}} \) and derive the Hartogs’ phenomenon. In Sect. 5, we construct the nilpotent Lie groups of step two associated with quadratic hypersurfaces. By constructing a diffeomorphism from the group \(\mathscr {H}\) to the hypersurface \(\mathcal {S}\) in (1.1), we show that the pushforward of the tangential k-Cauchy–Fueter operator on the group \(\mathscr {H} \) coincides with the restriction of the k-Cauchy–Fueter operator on \(\mathbb {H}^{n+1}\) to this hypersurface. Therefore, the restriction of a k-regular functions to \(\mathcal {S}\) is k-CF on \(\mathscr {H}.\)k-CF functions are abundant because so are k-regular functions on \(\mathbb {H}^{n+1}\) [21]. In Appendix, we give the expression of \(\Box _1\) for \(n=2,k=2.\)

2 The tangential k-Cauchy–Fueter complexes

2.1 The right quaternionic Heisenberg group \(\mathscr {H}\) and the locally flat compact manifold \(\mathscr {H}/\mathscr {H}_\mathbb {Z}\)

The multiplication of the right quaternionic Heisenberg group \(\mathscr {H}\) can be written in terms of real variables (cf. [36, (2.13)]) as

$$\begin{aligned} (x,t)\cdot ({y},{s})= {\left( x+y,t_{\beta }+s_{\beta }+2 \sum _{l=0}^{n-1}\sum _{j,k=1}^{4}B_{kj}^{\beta }x_{4l+k}y_{4l+j}\right) }, \end{aligned}$$
(2.1)

for \(x,y\in \mathbb {R}^{4n},\ t,s\in \mathbb {R}^{3},\ \beta =1,2,3,\) where \(B_{kj}^{\beta }\) is the (kj)th entry of the following matrices

$$\begin{aligned} \begin{aligned} B^{1}&:=\left( \begin{array}{cccc} 0 &{}\quad -\,1 &{}\quad 0 &{}\quad 0\\ 1&{}\quad 0&{}\quad 0&{}\quad 0\\ 0&{}\quad 0&{}\quad 0&{}\quad -\,1\\ 0 &{}\quad 0&{}\quad 1 &{}\quad 0\end{array}\right) , B^{2}:=\left( \begin{array}{cccc} 0 &{}\quad 0 &{}\quad -\,1 &{}\quad 0\\ 0&{}\quad 0&{}\quad 0&{}\quad 1\\ 1&{}\quad 0&{}\quad 0&{}\quad 0\\ 0 &{}\quad -\,1&{}\quad 0 &{}\quad 0\end{array}\right) , \\ B^{3}&:=\left( \begin{array}{cccc} 0 &{}\quad 0 &{}\quad 0 &{}\quad -\,1\\ 0&{}\quad 0&{}\quad -\,1&{}\quad 0\\ 0&{}\quad 1&{}\quad 0&{}\quad 0\\ 1 &{}\quad 0&{}\quad 0 &{}\quad 0\end{array}\right) , \end{aligned} \end{aligned}$$
(2.2)

satisfying the commutating relation of quaternions \((B^{1})^{2}=(B^{2})^{2}=(B^{3})^{2}=-id,\ B^{1}B^{2}=B^{3}.\) This is because for \(x=x_{1}+x_{2}{} \mathbf i +x_{3}{} \mathbf j +x_{4}{} \mathbf k \) and \(x'=x'_{1}+x'_{2}{} \mathbf i +x'_{3}{} \mathbf j +x'_{4}{} \mathbf k ,\) we have

$$\begin{aligned} \begin{aligned} \mathrm{Im}(x\overline{x'})&=(-x_{1}x'_{2}+x_{2}x'_{1}-x_{3}x'_{4}+x_{4}x'_{3})\mathbf i +(-x_{1}x'_{3}+x_{3}x'_{1}+x_{2}x'_{4}-x_{4}x'_{2})\mathbf j \\&\quad +\,(-x_{1}x'_{4}+x_{4}x'_{1}-x_{2}x'_{3}+x_{3}x'_{2})\mathbf k =\sum _{\beta =1}^{3}\sum _{k,j=1}^{4}B_{kj}^{\beta }x_{k}x'_{j}{} \mathbf i _{\beta }, \end{aligned} \end{aligned}$$

where \(\mathbf i _0=1,\mathbf i _1=\mathbf i , \mathbf i _2=\mathbf j ,\mathbf i _3=\mathbf k \). For fixed point \((y,s)\in \mathscr {H},\) the left translate\(\tau _{(y,s)}:\mathscr {H}\longrightarrow \mathscr {H},\)\( (x,t)\longmapsto (y,s)\cdot (x,t),\) is an affine transformation given by a lower triangular matrix by (2.1). So the Lebesgue measure on \(\mathbb {R}^{4n+3}\) is an invariant measure under the left translation of \(\mathscr {H}.\) Recall that we have the following left invariant vector fields on \(\mathscr {H} \):

$$\begin{aligned} (Y_{a}f)(y ,s )=\left. \frac{\hbox {d}}{\hbox {d}t}f((y ,s )(te_{a},0)) \right| _{t=0} , \qquad a=1,2,\ldots ,4n, \end{aligned}$$
(2.3)

where \(e_{a}\) is \((0,\ldots ,1,\ldots ,0)\) with only the ath entry equal to 1. Then,

$$\begin{aligned} Y_{4l+j}:=\frac{\partial }{\partial y_{4l+j}}+2\sum _{\beta =1}^{3}\sum _{k=1}^{4}B^{\beta }_{kj}y_{4l+k} \frac{\partial }{\partial s_{\beta }}, \end{aligned}$$
(2.4)

whose brackets are

$$\begin{aligned}{}[Y_{4l+k},Y_{4l+j}] =4 \sum _{\beta =1}^{3}B_{kj}^{\beta }\partial _{s_{\beta }}, \quad \mathrm{and }\quad [Y_{4l+k},Y_{4l'+j}] =0\quad \mathrm{for}\ l\ne l', \end{aligned}$$
(2.5)

where \(l,l'=0,1,\ldots ,n-1,\)\(j,k=1,\ldots ,4.\) The SubLaplacian is defined as

$$\begin{aligned} \Delta _b:=-\sum _{a=1}^{4n}Y_{a}^2. \end{aligned}$$
(2.6)

The norm of the right quaternionic Heisenberg group \(\mathscr {H}\) is defined by

$$\begin{aligned} \Vert (y,{s})\Vert :=(|y|^{4}+|{s}|^{2})^{\frac{1}{4}}. \end{aligned}$$
(2.7)

Define balls \(B(\xi ,r):=\{\eta \in \mathscr {H} ;\Vert \xi ^{-1}\cdot \eta \Vert <r\}\) for \(\xi \in \mathscr {H}, r>0.\) The fundamental set of \(\mathscr {H}\) under the action of the lattice \(\mathscr {H}_{\mathbb {Z}}\) in (1.11) is

$$\begin{aligned} \mathscr {F}=\{\left. (y,s) \in \mathscr {H}\right| 0\le y_a<1,0\le s_\beta <1,a=1,\ldots ,4n,\beta =1,2,3\}. \end{aligned}$$
(2.8)

\(\mathscr {H}/\mathscr {H}_{\mathbb {Z}}\) is equivalent to \(\mathscr {F}\) as a set.

Proposition 2.1

\(\mathscr {H}\) is the disjoint union of \(\tau _{(n,m)}\mathscr {F}\) with \((n,m)\in \mathscr {H}_{\mathbb {Z}}.\)

Proof

We need to prove that for any \((y,s)\in \mathscr {H},\) there exist unique \((y',s')\in \mathscr {F}\) and \((n,m)\in \mathscr {H}_{\mathbb {Z}}\) such that \((y,s)=(n,m)\cdot (y',s').\) Let \( (n_a,m_a)\in \mathscr {H}_{\mathbb {Z}},a=1,2.\) By the multiplication law (2.1), we have

$$\begin{aligned} \begin{aligned} (n_a,m_a)\cdot ({y},{s})=\left( n_a+y,(m_a)_\beta +s_{\beta }+2 \sum _{l=0}^{n-1}\sum _{j,k=1}^{4}B_{kj}^{\beta }(n_a)_{4l+k}y_{4l+j}\right) . \end{aligned} \end{aligned}$$
(2.9)

If \(n_1\ne n_2,\) the y-coordinates of \((n_1,m_1)\cdot ({y},{s})\) and \((n_2,m_2)\cdot ({y},{s})\) are \(n_1+y\) and \(n_2+y,\) respectively, which are different. If \(n_1= n_2,m_1\ne m_2,\) we see that their s-coordinates in (2.9) must be different. This proves the uniqueness.

For \((y,s)=(y_1,\ldots ,y_{4n},s_1,s_2,s_3),\) we can choose \(y'\in \mathbb {R}^{4n}\) with \(0\le y_j'<1 \) and \(n\in \mathbb {Z}^{4n}\) such that \(y_j={n}_j+y_j' \). Then, we can determine \(s'\in \mathbb {R}^{3}\) and \(m\in \mathbb {Z}^{3}\) satisfying

$$\begin{aligned} m_{\beta }+s'_\beta =s_\beta -2 \sum _{l=0}^{n-1}\sum _{j,k=1}^{4n}B_{kj}^{\beta }n_{4l+k}y'_{4l+j},\ \ \mathrm{with}\ 0\le s_\beta '<1, \end{aligned}$$

for \(\beta =1,2,3.\) So \(\mathscr {H}\) is the disjoint union of \(\tau _{(n,m)}\mathscr {F}.\) The proposition is proved. \(\square \)

\(\mathscr {H}/\mathscr {H}_{\mathbb {Z}}\) has the structure of a locally flat manifold as follows (cf. [22, p. 238]). Let \(\pi :\mathscr {H}\rightarrow \mathscr {H}/\mathscr {H}_{\mathbb {Z}}\) be the projection. We can find a finite number of balls \(B(\xi _j,r),\ j=1,\ldots ,N,\) covering \(\mathscr {F}\) with r sufficiently small so that \(\tau _{(n,m)} B(\xi _j,r)\cap B(\xi _j,r)=\emptyset \) for any \((0,0)\ne (n,m)\in \mathscr {H}_{\mathbb {Z}}.\) Note that \(\pi B(\xi _i,r)\cap \pi B(\xi _j,r)\ne \emptyset \) for \(i\ne j\) if and only if there exist unique \({(n,m)}\in \mathscr {H}_{\mathbb {Z}},\) such that

$$\begin{aligned} \tau _{(n,m)}B(\xi _i,r)\cap B(\xi _j,r)\ne \emptyset . \end{aligned}$$
(2.10)

Then, we can construct coordinates charts \((\pi B(\xi _j,r),\phi _j),\) where \(\phi _j:\pi B(\xi _j,r)\rightarrow B(\xi _j,r)\) and the transition function \(\phi _j\circ \phi _i^{-1}\) is given by \(\tau _{(n,m)}\) for some \((n,m)\in \mathscr {H}_{\mathbb {Z}}\) such that (2.10) holds.

A function is called periodic on \(\mathscr {H}\) if

$$\begin{aligned} f(y,s)=f((n,m)(y,s)) \end{aligned}$$

for any \((n,m)\in \mathscr {H}_{\mathbb {Z}}.\) A function over \(\mathscr {H}/\mathscr {H}_{\mathbb {Z}}\) can be viewed as a function on \(\mathscr {F}\) and be extended to a periodic function on \(\mathscr {H}\) by

$$\begin{aligned} {f}(y,s)=f((n,m)\cdot (y',s'))=f(y',s'), \end{aligned}$$
(2.11)

for \((y,s)=(n,m)\cdot (y',s')\) and \((y',s')\in \mathscr {F}.\) If f is periodic, then so is \(Y_af\) for any a. This is because

$$\begin{aligned} \begin{aligned} (Y_{a}f)(y',s')&=\left. \frac{\hbox {d}}{\hbox {d}t}f((y',s')(te_{a},0)) \right| _{t=0}=\left. \frac{\hbox {d}}{\hbox {d}t}f((n,m)(y',s')(te_{a},0)) \right| _{t=0}\\&=(Y_{a}f)(y,s), \end{aligned} \end{aligned}$$

for \(e_{a} \) as in (2.3). Thus, the action of \(Y_a\) on functions over \(\mathscr {H}/\mathscr {H}_{\mathbb {Z}}\) is well-defined, i.e., it is a vector field over \(\mathscr {H}/\mathscr {H}_{\mathbb {Z}}.\)

2.2 Complex horizontal vector fields \(Z_A^{A'}\)’s and the tangential k-Cauchy–Fueter operator

We consider the following complex horizontal left invariant vector fields on \(\mathscr {H} \):

$$\begin{aligned} \left( Z_{AA'}\right) :=\left( \begin{array}{c@{\quad }c} Y_{1}+\mathbf i Y_{2}&{} -Y_{3}-\mathbf i Y_{4}\\ Y_{3}-\mathbf i Y_{4}&{}\ \ Y_{1}-\mathbf i Y_{2}\\ \ \ \ \ \ \vdots &{}\ \ \ \ \ \ \ \vdots \\ Y_{4l+1}+\mathbf i Y_{4l+2}&{} -Y_{4l+3}-\mathbf i Y_{4l+4}\\ Y_{4l+3}-\mathbf i Y_{4l+4}&{} \ \ Y_{4l+1}-\mathbf i Y_{4l+2}\\ \ \ \ \ \vdots &{}\ \ \ \ \ \ \ \vdots \end{array}\right) \end{aligned}$$
(2.12)

where \(A=0,1,\ldots ,2n-1,\)\(A'=0',1'.\) It is motivated by the embedding \(\tau \) of quaternionic algebra \(\mathbb {H}\) into \(\mathfrak {gl}(2,\mathbb {C}):\)

$$\begin{aligned} \tau (x_{1}+x_{2}{} \mathbf i +x_{3}{} \mathbf j +x_{4}{} \mathbf k )= \left( \begin{array}{r@{\quad }r} x_{1}+\mathbf i x_{2}&{} -x_{3}-\mathbf i x_{4}\\ x_{3}-\mathbf i x_{4}&{} x_{1}-\mathbf i x_{2}\end{array}\right) \end{aligned}$$
(2.13)

and vector fields

$$\begin{aligned} \left( \nabla _{AA'}\right) :=\left( \begin{array}{c@{\quad }c} \partial _{x_{1}}+\mathbf i \partial _{x_{2}}&{} -\partial _{x_{3}}-\mathbf i \partial _{x_{4}}\\ \partial _{x_{3}}-\mathbf i \partial _{x_{4}}&{}\ \ \partial _{x_{1}}-\mathbf i \partial _{x_{2}}\\ \ \ \ \ \ \vdots &{}\ \ \ \ \ \ \ \vdots \\ \partial _{x_{4l+1}}+\mathbf i \partial _{x_{4l+2}}&{} -\partial _{x_{4l+3}}-\mathbf i \partial _{x_{4l+4}}\\ \partial _{x_{4l+3}} -\mathbf i \partial _{x_{4l+4}}&{} \ \ \partial _{x_{4l+1}}-\mathbf i \partial _{x_{4l+2}}\\ \ \ \ \ \vdots &{}\ \ \ \ \ \ \ \vdots \end{array}\right) \end{aligned}$$
(2.14)

to construct the k-Cauchy–Fueter operators on \(\mathbb {H}^{n+1}\) in [35]. We will use matrices

$$\begin{aligned} (\varepsilon _{A'B'})=\left( \begin{array}{c@{\quad }c} 0&{}\quad 1\\ -\,1&{}\quad 0\end{array}\right) ,\quad (\varepsilon ^{A'B'})=\left( \begin{array}{c@{\quad }c} 0&{}\quad -\,1\\ 1&{}\quad 0\end{array}\right) \end{aligned}$$
(2.15)

to raise or lower primed indices, e.g., \(Z_{A}^{A'}=\sum _{B'=0',1'} Z_{AB'}\varepsilon ^{B'A'}.\) Here, \((\varepsilon ^{A'B'})\) is the inverse of \((\varepsilon _{A'B'}).\) Then,

$$\begin{aligned} Z_A^{0'}=Z_{A1'},\quad Z_A^{1'}=-Z_{A0'}, \end{aligned}$$

and

$$\begin{aligned} \left( Z_A^{A'}\right) =\left( \begin{array}{c@{\quad }c} \vdots &{} \vdots \\ Z_{2l}^{0'}&{} Z_{2l}^{1'}\\ Z_{2l+1}^{0'}&{} Z_{2l+1}^{1'}\\ \vdots &{} \vdots \end{array}\right) =\left( \begin{array}{cc} \vdots &{} \vdots \\ -Y_{4l+3}-\mathbf i Y_{4l+4}&{}-Y_{4l+1}-\mathbf i Y_{4l+2}\\ Y_{4l+1}-\mathbf i Y_{4l+2}&{} -Y_{4l+3}+\mathbf i Y_{4l+4}\\ \vdots &{} \vdots \end{array}\right) . \end{aligned}$$
(2.16)

An element of \(\mathbb {C}^{2}\) is denoted by \((f_{A'})\) with \(A'=0',1'.\) The symmetric power \(\odot ^{p}\mathbb {C}^{2}\) is a subspace of \(\otimes ^{p}\mathbb {C}^2,\) whose element is a \(2^p\)-tuple \((f_{A'_{1}A'_2\ldots A'_{p}})\) with \(A'_{1},A'_2,\ldots ,A'_{p}=0',1',\) such that \(f_{A'_{1}A'_2\ldots A'_{p}}\in \mathbb {C}\) are invariant under permutations of subscripts, i.e.,

$$\begin{aligned} f_{A'_{1}A'_2\ldots A'_{p}}=f_{A'_{\sigma (1)}A'_{\sigma (2)}\ldots A'_{\sigma (p)}} \end{aligned}$$

for any \(\sigma \) in the group \(S_p\) of permutations of p letters. An element of \(\odot ^{p}\mathbb {C}^2\otimes \wedge ^q\mathbb {C}^{2n}\) is given by a tuple \((f_{A_1'\ldots A_p'A_1 \ldots A_q})\in (\otimes ^{p}\mathbb {C}^2)\otimes (\otimes ^q\mathbb {C}^{2n}),\) which is invariant under permutations of subscripts of \(A_1',\ldots ,A_p',\) and antisymmetric under permutations of subscripts of \(A_1,\ldots ,A_q =0,1,\ldots 2n-1.\) In the sequel, we will write \(f_{A A_2'A_3'\ldots A_k'}:=f_{A_2'A_3'\ldots A_k'A}\) and \(f_{A_3'\ldots A_k'AB}:=f_{ABA_3'\ldots A_k'}\) for convenience. We will use symmetrization of primed indices

$$\begin{aligned} f_{\ldots (A_1'\ldots A_p')\ldots }:=\frac{1}{p!}\sum _{\sigma \in S_p}f_{\ldots A_{\sigma (1)}'\ldots A_{\sigma (p)}'\ldots }. \end{aligned}$$
(2.17)

The tangential k-Cauchy–Fueter operator in (1.4) is given by

$$\begin{aligned} \begin{aligned} (\mathscr {D}_{0}f)_{AA'_2\ldots A'_k}:=\sum _{A'_1=0',1'}Z_{A}^{A'_1}f_{A'_1 A'_2\ldots A'_k}, \end{aligned} \end{aligned}$$
(2.18)

for \(f\in C^1(\Omega ,\mathscr {V}_0) \). The k-Cauchy–Fueter operator on \(\mathbb {H}^{n+1}\) [35] is \(\widehat{\mathscr {D}}_0: C^1(\mathbb {H}^{n+1},\mathscr {V}_0)\rightarrow C^1(\mathbb {H}^{n+1},\mathscr {V}_1)\) with

$$\begin{aligned} \left( \widehat{\mathscr {D}}_{0}f\right) _{A_2'\ldots A_k'A}:=\sum _{B'=0',1'}\nabla _{A}^{B'}f_{B' A_2'\ldots A_k'}, \end{aligned}$$

where \(\nabla \) is given by (2.14). A \(\mathscr {V}_0\)-valued distribution f is called k-regular on \(\Omega \in \mathbb {H}^{n+1}\) if \(\widehat{\mathscr {D}}_0f=0\) on \(\Omega \) in the sense of distributions.

2.3 Commutators of complex horizontal vector fields

The following nice behavior of commutators of \(Z_A^{A'}\)’s plays a very important role to show that (1.4) is a complex and to establish the \(L^2\) estimate. It is also the reason why the tangential k-Cauchy–Fueter complex on the right Heisenberg group is simpler than that on the left one.

Lemma 2.1

  1. (1)

    Vector fields in each column in (2.16) are commutative, i.e., for fixed \(A'=0'\ \mathrm{or}\ 1',\)

    $$\begin{aligned}{}[Z_A^{A'},Z_B^{A'}]=0, \end{aligned}$$
    (2.19)

    for any \(A,B=0,\ldots ,2n-1.\)

  2. (2)

    We have

    $$\begin{aligned} \begin{aligned}&[Z_{2l}^{0'},Z_{2l}^{1'}]= \overline{[Z_{2l+1}^{0'},Z_{2l+1}^{1'}]} =8\left( \partial _{s_2}+\mathbf {i}\partial _{s_3}\right) ,\\&[Z_{2l}^{0'},Z_{2l+1}^{1'}] =[Z_{2l+1}^{0'},Z_{2l}^{1'}]=8\mathbf {i}\partial _{s_1}, \end{aligned} \end{aligned}$$
    (2.20)

    \(l=0,\ldots ,n-1,\) and any other bracket vanishes.

Proof

(1):

If \(\{A,B\}\ne \{2l,2l+1\}\) for any integer l,  we have

$$\begin{aligned}{}[Z_A^{A'},Z_B^{B'}]=0,\quad \mathrm{for}\ A',B'=0',1', \end{aligned}$$

by using (2.5) because \(Z_A^{A'}\) and \(Z_B^{B'}\) only involve \(Y_{4l+j}\)’s for different l. It follows from (2.2) (2.5) that

$$\begin{aligned} \begin{aligned} \ [Y_{4l+1},Y_{4l+2}]&=\ \ [Y_{4l+3},Y_{4l+4}]=-\,4\partial _{s_1},\\ [Y_{4l+1},Y_{4l+3}]&=-\,[Y_{4l+2},Y_{4l+4}]=-\,4\partial _{s_2},\\ [Y_{4l+1},Y_{4l+4}]&=\ \ [Y_{4l+2},Y_{4l+3}]=-\,4\partial _{s_3}. \end{aligned} \end{aligned}$$
(2.21)

Then, for \(\{A,B\}=\{2l,2l+1\},\) we have

$$\begin{aligned} \begin{aligned} \ [Z_{2l}^{0'},Z_{2l+1}^{0'}]&=[-Y_{4l+3}- \mathbf {i}Y_{4l+4},Y_{4l+1}-\mathbf {i}Y_{4l+2}]\\&=[Y_{4l+1},Y_{4l+3}]+[Y_{4l+2},Y_{4l+4}]-\mathbf {i}[Y_{4l+2}, Y_{4l+3}]+\mathbf {i}[Y_{4l+1},Y_{4l+4}]=0,\\ \ [Z_{2l}^{1'},Z_{2l+1}^{1'}]&=[-Y_{4l+1}-\mathbf {i}Y_{4l+2},-Y_{4l+3} +\mathbf {i}Y_{4l+4}]\\&=[Y_{4l+1},Y_{4l+3}]+[Y_{4l+2},Y_{4l+4}]+\mathbf {i}[Y_{4l+2}, Y_{4l+3}]-\mathbf {i}[Y_{4l+1},Y_{4l+4}]=0, \end{aligned} \end{aligned}$$

by (2.21). Then, (2.19) follows.

(2):

Similarly, we have

$$\begin{aligned} \begin{aligned} \ [Z_{2l}^{0'},Z_{2l}^{1'}]&=[-Y_{4l+3}-\mathbf {i} Y_{4l+4},-Y_{4l+1}-\mathbf {i}Y_{4l+2}]\\&=-\,[Y_{4l+1},Y_{4l+3}]+[Y_{4l+2},Y_{4l+4}]\\&\quad -\,\mathbf {i}[Y_{4l+2}, Y_{4l+3}]-\mathbf {i}[Y_{4l+1},Y_{4l+4}]= 8(\partial _{s_{2}}+\mathbf {i}\partial _{s_{3}}),\\ \ [Z_{2l+1}^{0'},Z_{2l+1}^{1'}]&=\overline{[Z_{2l}^{0'},Z_{2l}^{1'}]} =8(\partial _{s_{2}}-\mathbf {i}\partial _{s_{3}}),\\ \ [Z_{2l}^{0'},Z_{2l+1}^{1'}]&=[-Y_{4l+3}- \mathbf {i}Y_{4l+4},-Y_{4l+3}+\mathbf {i}Y_{4l+4}] =-\,2\mathbf {i}[Y_{4l+3},Y_{4l+4}] =8\mathbf {i}\partial _{s_{1}},\\ \ [Z_{2l+1}^{0'},Z_{2l}^{1'}]&=[Y_{4l+1}-\mathbf {i}Y_{4l+2},-Y_{4l+1}- \mathbf {i}Y_{4l+2}] =-\,2\mathbf {i}[Y_{4l+1},Y_{4l+2}] =8\mathbf {i}\partial _{s_{1}}, \end{aligned} \end{aligned}$$

by (2.21). The lemma is proved. \(\square \)

On the left quaternionic Heisenberg group, vector fields in each column are not commutative (2.43)–(2.44). We have the following corollary directly by Lemma 2.1 (2).

Corollary 2.1

$$\begin{aligned}{}[Z_{A}^{0'},Z_{B}^{1'}]+[Z_{A}^{1'},Z_{B}^{0'}]=0, \end{aligned}$$
(2.22)

for any \(A,B=0,\ldots ,2n-1.\)

2.4 The tangential k-Cauchy–Fueter complex

Differential operators in the complex (1.4) are as follows. For \(j=0,1,\ldots ,k-1,\)\(\mathscr {D}_j:C^{\infty }(\Omega ,\mathscr {V}_j)\rightarrow C^{\infty }(\Omega ,\mathscr {V}_{j+1})\) with \(\mathscr {V}_j=\odot ^{k-j}\mathbb {C}^2\otimes \wedge ^j\mathbb {C}^{2n} \) is a differential operator of the first order given by

$$\begin{aligned} \left( \mathscr {D}_jf\right) _{A_0\ldots A_jA_1'\ldots A_{k-j-1}'}=(j+1)\sum _{A'=0',1'}Z_{[A_0}^{A'}f_{A_1\ldots A_j] A'A_1'\ldots A_{k-j-1}'}, \end{aligned}$$
(2.23)

where \([A_0A_1\ldots A_j]\) is the antisymmetrization of unprimed indices given by

$$\begin{aligned} f_{\ldots [A_1\ldots A_p]\ldots }:=\frac{1}{p!}\sum _{\sigma \in S_p}\mathrm{sign}(\sigma )f_{\ldots A_{\sigma (1)}\ldots A_{\sigma (p)}\ldots }. \end{aligned}$$
(2.24)

In particular, \(h_{[AB]}:=\frac{1}{2}(h_{AB}-h_{BA})\). By definition, we have

$$\begin{aligned} f_{\ldots [A_1\ldots [A_j\ldots A_l]\ldots A_p]\ldots }=f_{\ldots [A_1\ldots A_j\ldots A_l \ldots A_p]\ldots }. \end{aligned}$$
(2.25)

\(\mathscr {D}_k:C^{\infty }(\Omega ,\mathscr {V}_k)\rightarrow C^{\infty }(\Omega ,\mathscr {V}_{k+1})\) with \(\mathscr {V}_k=\wedge ^k\mathbb {C}^{2n}\) and \(\mathscr {V}_{k+1}=\wedge ^{k+2}\mathbb {C}^{2n}\) is a differential operator of the second order given by

$$\begin{aligned} \left( \mathscr {D}_{k}f\right) _{A_1\ldots A_{k+2}}=(k+2)Z_{[A_1}^{0'}Z_{A_2}^{1'}f_{A_3\ldots A_{k+2}]}. \end{aligned}$$
(2.26)

For \(j=k+1,\ldots ,2n-2,\)\(\mathscr {D}_j:C^{\infty }(\Omega ,\mathscr {V}_j)\rightarrow C^{\infty }(\Omega ,\mathscr {V}_{j+1})\) with \(\mathscr {V}_j=\odot ^{j-k-1}\mathbb {C}^2\otimes \wedge ^{j+1}\mathbb {C}^{2n} \) is a differential operator of the first order given by

$$\begin{aligned} \left( \mathscr {D}_{j}f\right) _{A_1\ldots A_{j+2}}^{A_1'\ldots A_{j-k}'}=(j+2)Z_{[A_1}^{(A_1'}f^{A_2'\ldots A'_{{j-k}})}_{A_2\ldots A_{j+2}]}. \end{aligned}$$
(2.27)

Remark 2.1

The k-Cauchy–Fueter complex on \(\mathbb {H}^n\) [35, 41] is the same as (1.4)–(1.5) with \({\mathscr {H}}\) replaced by \(\mathbb {H}^n\) and \(Z_{A}^{A'}\) in definition of \(\mathscr {D}_{j}\)’s in (2.23), (2.26) and (2.27) replaced by \(\nabla _{A}^{A'}\) in (2.14).

Lemma 2.2

$$\begin{aligned} Z_{[A}^{(A'}Z_{B]}^{B')}=0, \end{aligned}$$
(2.28)

for any \(A,B=0,\ldots ,2n-1\) and \(A',B'=0',1'.\)

Proof

Note that

$$\begin{aligned} 2Z_{[A}^{A'}Z_{B]}^{A'}=Z_{A}^{A'}Z_{B}^{A'}-Z_{B}^{A'}Z_{A}^{A'}=[Z_{A}^{A'}, Z_{B}^{A'}]=0, \end{aligned}$$
(2.29)

by (2.19), and

$$\begin{aligned} \begin{aligned} 4 Z_{[A}^{(0'}Z_{B]}^{1')}&= 2Z_{[A}^{0'}Z_{B]}^{1'}+2Z_{[A}^{1'}Z_{B]}^{0'}=Z_{A}^{0'}Z_{B}^{1'}- Z_{B}^{0'}Z_{A}^{1'}+Z_{A}^{1'}Z_{B}^{0'}- Z_{B}^{1'}Z_{A}^{0'}\\ {}&=[Z_{A}^{0'},Z_{B}^{1'}]+[Z_{A}^{1'},Z_{B}^{0'}]=0, \end{aligned} \end{aligned}$$

by Corollary 2.1. The lemma is proved. \(\square \)

Now, let us check (1.4) to be a complex by direct calculation as in [41, Section 3.1].

Theorem 2.1

(1.4) is a complex, i.e.,

$$\begin{aligned} \mathscr {D}_{j+1}\circ \mathscr {D}_j=0 \end{aligned}$$
(2.30)

for each j.

Proof

For \(A,B=0,\ldots , 2n-1\) and \(A_3',\ldots , A_k'=0',1',\) we have

$$\begin{aligned} \begin{aligned} (\mathscr {D}_1\circ \mathscr {D}_0f)_{ABA_3'\ldots A_k'}&=2\sum _{A'=0',1'} Z_{[A}^{A'}(\mathscr {D}_0f)_{B]A'A_3'\ldots A_k'} =2\sum _{A',C'=0',1'} Z_{[A}^{A'}Z_{B]}^{C'}f_{C'A'A_3'\ldots A_k'}\\&= 2\sum _{A',C'=0',1'} Z_{[A}^{(A'}Z_{B]}^{C')}f_{C'A'A_3'\ldots A_k'}=0, \end{aligned} \end{aligned}$$

by Lemma 2.2 and \(f_{C'A'A_3'\ldots A_k'}=f_{A'C'A_3'\ldots A_k'}\). For general \(j=1,\ldots ,k-2,\) we have

$$\begin{aligned} \begin{aligned}&(\mathscr {D}_{j+1}\circ \mathscr {D}_jf)_{A_1\ldots A_{j+2}A_1'\ldots A_{k-j-2}'}\\&\quad =(j+2)(j+1)\sum _{A',C'=0',1'} Z_{[A_1}^{A'}Z_{[A_2}^{C'}f_{A_3\ldots A_{j+2}]]C'A'A_1'\ldots A_{k-j-2}'} \\&\quad = (j+2)(j+1) \sum _{A',C'=0',1'} Z_{[[A_1}^{(A'}Z_{A_2]}^{C')}f_{A_3\ldots A_{j+2}]C'A'A_1'\ldots A_{k-j-2}'}=0, \end{aligned} \end{aligned}$$

by using (2.25) repeatedly, Lemma 2.2 and f symmetric in the primed indices again.

For \(j=k-1,\) we have

$$\begin{aligned} \begin{aligned} (\mathscr {D}_{k}\circ \mathscr {D}_{k-1}f)_{A_1\ldots A_{k+2}}=&(k+2)k\sum _{A'=0',1'} Z_{[A_1}^{0'}Z_{A_2}^{1'}Z_{[A_3}^{A'}f_{A_4\ldots A_{k+2}]]A'}=0. \end{aligned} \end{aligned}$$

This is because if \(A'=1',\)\(Z_{[A_1}^{0'}Z_{[A_2}^{1'}Z_{A_3]}^{1'}f_{A_4\ldots A_{k+2}]1'}=0\) by using (2.29), and if \(A'=0',\)

$$\begin{aligned} Z_{[A_1}^{0'}Z_{A_2}^{1'}Z_{A_3]}^{0'}=Z_{[A_1}^{0'}Z_{[A_2}^{1'}Z_{A_3]]}^{0'}= -Z_{[A_1}^{0'}Z_{[A_2}^{0'}Z_{A_3]]}^{1'}= -Z_{[[A_1}^{0'}Z_{A_2]}^{0'}Z_{A_3]}^{1'}=0, \end{aligned}$$

by using (2.25) repeatedly and Corollary 2.1.

For \(j=k,\) we have

$$\begin{aligned} \left( \mathscr {D}_{k+1}\circ \mathscr {D}_kf\right) _{A_1\ldots A_{k+3}}^{A'}=(k+3)(k+2)Z_{[A_1}^{A'}Z_{[A_2}^{0'}Z_{A_3}^{1'}f_{A_4\ldots A_{k+3}]]}=0. \end{aligned}$$
(2.31)

This is because if \(A'=0',\)\(Z_{[[A_1}^{0'}Z_{A_2]}^{0'}Z_{A_3}^{1'}f_{A_4\ldots A_{k+3}] }=0\) by using (2.29), and if \(A'=1',\)

$$\begin{aligned} Z_{[A_1}^{1'}Z_{A_2}^{0'}Z_{A_3]}^{1'}=Z_{[A_1}^{1'}Z_{[A_2}^{0'}Z_{A_3]]}^{1'}= -Z_{[A_1}^{1'}Z_{[A_2}^{1'}Z_{A_3]]}^{0'}= -Z_{[[A_1}^{1'}Z_{A_2]}^{1'}Z_{A_3]}^{0'}=0, \end{aligned}$$

by using (2.25) repeatedly and Corollary 2.1.

For \(j=k+1,\ldots ,2n-2,\) we have

$$\begin{aligned} \left( \mathscr {D}_{j+1}\circ \mathscr {D}_jf\right) _{A_1\ldots A_{j+3}}^{A_1'\ldots A'_{j-k+1}}=(j+3)(j+2)Z^{((A_1'}_{[[A_1}Z_{A_2]}^{A_2')}f^{A_3'\ldots A'_{{j-k+1}})}_{A_3\ldots A_{j+3}]}=0, \end{aligned}$$

by Lemma 2.2. The theorem is proved. \(\square \)

2.5 Comparison with the left case

Recall that a transformation T on \(\mathscr {H}\) is called conformal if \(\Vert T_*W_1\Vert =\Vert T_*W_2\Vert \) for any two horizontal vector fields \(W_1\) and \(W_2\) with \(\Vert W_1\Vert =\Vert W_2\Vert ,\) where \(\Vert W\Vert ^2:= \sum _{j=1}^{4n}a_j^2 \) if we write \(W=\sum _{j=1}^{4n}a_jY_j.\) It is known that the group of conformal transformations on \(\mathscr {H}\) is the real semisimple Lie group \(\mathrm{Sp}(n+1,1)\) of rank one (cf., e.g., [18]) generated by the following transformations:

  1. (1)

    Dilations:

    $$\begin{aligned} D_{\delta }:(y,s)\longrightarrow (\delta y,\delta ^{2}s),\ \delta >0; \end{aligned}$$
    (2.32)
  2. (2)

    Left translations:

    $$\begin{aligned} \tau _{(x,{t})}:(y,{s})\longrightarrow (x,{t})\cdot (y,{s}); \end{aligned}$$
    (2.33)
  3. (3)

    Rotations:

    $$\begin{aligned} R_{\mathbf {a}}:(y,s)\longrightarrow (y{\mathbf {a}},s),\ \mathrm{for} \ {\mathbf {a}}\in \mathrm{Sp}(n), \end{aligned}$$
    (2.34)

    where

    $$\begin{aligned} \mathrm{Sp}(n)=\{{\mathbf {a}}\in \mathrm{GL}(n,\mathbb {H})|{{\mathbf {a}}\bar{{\mathbf {a}}}^{t}}=I_{n}\}; \end{aligned}$$
  4. (4)

    The inversion:

    $$\begin{aligned} R:(y,s)\longrightarrow \left( -\,(|y|^{2}-s)^{-1}y, \frac{-s}{|y|^{4}+|s|^{2}}\right) ; \end{aligned}$$
    (2.35)
  5. (5)

    \(\mathrm{Sp}(1)\) acts on \(\mathscr {H}\) as

    $$\begin{aligned} \sigma :(y,s)\longrightarrow (\sigma y,\sigma {s}\sigma ^{-1}), \end{aligned}$$
    (2.36)

    where the action on the first factor is left multiplication by \(\sigma \in \mathbb {H}\) with \(|\sigma |=1,\) while the action on the second factor is isomorphism with \(\mathrm{SO}(3)\).

It is known that \(\mathrm{Sp}(n+1,1)\) is a real form of \(\mathrm{Sp}(2(n+2),\mathbb {C})\), whose Lie algebra \(\mathfrak {g}=\mathfrak {sp}(2(n+2),\mathbb {C})\) has the decomposition \(\mathfrak {g}=\mathfrak {g}_{-2}\oplus \mathfrak {g}_{-1}\oplus \mathfrak {g}_{0} \oplus \mathfrak {g}_{1}\oplus \mathfrak {g}_{2},\) where \(\mathfrak {g}_{-2}\) is an complex abelian subalgebra generated by \(T_1,T_2,T_3,\) and \(\mathfrak {g}_{-1}\) is generated by \(\{Y_{AA'}\},A=0,1,\ldots ,2n-1,A'=0',1'\) with

$$\begin{aligned} \begin{aligned} \ [Y_{A0'},Y_{(n+A)0'}]&=4T_2,\\ \ [Y_{A1'},Y_{(n+A)1'}]&=4T_3,\\ \ [Y_{A0'},Y_{(n+A)1'}]&=[Y_{A1'},Y_{(n+A)0'}]=4T_1, \end{aligned} \end{aligned}$$
(2.37)

and any other bracket vanishes (cf. [37, (2.10)]). \({\mathfrak {p}}:=\mathfrak {g}_{0} \oplus \mathfrak {g}_{1}\oplus \mathfrak {g}_{2}\) is a parabolic subgroup. \(\mathfrak {u}_-:=\mathfrak {g}_{-2}\oplus \mathfrak {g}_{-1}.\) Then,

$$\begin{aligned} \mathfrak {g}=\mathfrak {u}_-\oplus {\mathfrak {p}}. \end{aligned}$$
(2.38)

Let \(\mathrm{U}_-\) be the complex Lie group with Lie algebra \(\mathfrak {u}_-\). There exist exact sequences [37, Theorem 3.2.1] on \(\mathrm{U}_-\)

$$\begin{aligned} \begin{aligned} 0&\rightarrow \mathcal {R}\left( \mathrm{U}_-,\odot ^{k}\mathbb {C}^2\right) \xrightarrow {Q_{0}^{(k)}} \mathcal {R}\left( \mathrm{U}_-,\odot ^{k-1}\mathbb {C}^2\otimes {V}^{(1)}\right) \xrightarrow {Q_{1}^{(k)}}\ldots \rightarrow \mathcal {R}\left( \mathrm{U}_-,{V}^{(k)}\right) \\&\xrightarrow {Q_{k}^{(k)}} \mathcal {R}\left( \mathrm{U}_-,{V}^{(k+2)}\right) \xrightarrow {Q_{k+1}^{(k)}}\ldots \xrightarrow {Q_{2n-1}^{(k)}}\mathcal {R}\left( \mathrm{U}_-,\odot ^{2n-k} \mathbb {C}^2\right) \rightarrow 0, \end{aligned} \end{aligned}$$
(2.39)

for \(0\le k\le n-2,\) where operators \(Q_{j}^{(k)}\)’s are defined in terms of \(Y_{AA'},T_\beta \) (cf. [37, Theorem 1.0.1]). Here, \(V^{(j)}\) is the irreducible representation of \(\mathfrak {sp}(2n,\mathbb {C})\) with the highest weight to be the jth fundamental weight \(\omega _j\) and \(\mathcal {R}( \mathrm{U}_-,V)\) is the space of V-valued polynomials over \(\mathrm{U}_-.\) These complexes are constructed by twistor method, and operators \(Q_{j}^{(k)}\)’s are invariant under \(\mathrm {Sp}(2(n+2),\mathbb {C}).\)

The multiplication (1.6) of the left quaternionic Heisenberg group \(\widetilde{{\mathscr {H}}}\) can be written as

$$\begin{aligned} (x,t)\cdot ({y},{s})= {\left( x+y,t_{\beta }+s_{\beta }+2 \sum _{l=0}^{n-1}\sum _{j,k=1}^{4}I_{kj}^{\beta }x_{4l+k}y_{4l+j}\right) }, \end{aligned}$$
(2.40)

for \(x,y\in \mathbb {R}^{4n},\ t,s\in \mathbb {R}^{3},\, \beta =1,2,3,\) where \(I_{kj}^{\beta }\) is the (kj)th entry of the following matrices

$$\begin{aligned} \begin{aligned} I^{1}&:=\left( \begin{array}{c@{\quad }c@{\quad }c@{\quad }c} 0 &{}\quad 1 &{}\quad 0 &{}\quad 0\\ -\,1&{}\quad 0&{}\quad 0&{}\quad 0\\ 0&{}\quad 0&{}\quad 0&{}\quad -\,1\\ 0 &{}\quad 0&{}\quad 1 &{}\quad 0 \end{array}\right) , I^{2}:=\left( \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }} 0 &{}\quad 0 &{}\quad 1 &{}\quad 0\\ 0&{}\quad 0&{}\quad 0&{}\quad 1\\ -\,1&{}\quad 0&{}\quad 0&{}\quad 0\\ 0 &{}\quad -\,1&{}\quad 0 &{}\quad 0 \end{array}\right) , \\ I^{3}&:=\left( \begin{array}{c@{\quad }c@{\quad }c@{\quad }c} 0 &{}\quad 0 &{}\quad 0 &{}\quad 1\\ 0&{}\quad 0&{}\quad -\,1&{}\quad 0\\ 0&{}\quad 1&{}\quad 0&{}\quad 0\\ -\,1 &{}\quad 0&{}\quad 0 &{}\quad 0 \end{array}\right) , \end{aligned} \end{aligned}$$
(2.41)

satisfying the commutating relation of quaternions. Note that

$$\begin{aligned} \widetilde{X}_{4l+j}=\frac{\partial }{\partial x_{4l+j}}+2\sum _{\beta =1}^{3}\sum _{k=1}^{4}I^{\beta }_{kj}x_{4l+k} \frac{\partial }{\partial t_{\beta }} \end{aligned}$$
(2.42)

is standard left invariant vector field on \(\widetilde{\mathscr {H}}.\) Denote

$$\begin{aligned} \left( \widetilde{Z}_{AA'}\right) :=\left( \begin{array}{cc} \widetilde{X}_{1}+\mathbf {i}\widetilde{X}_{2}&{} -\widetilde{X}_{3}-\mathbf {i}\widetilde{X}_{4}\\ \widetilde{X}_{3}-\mathbf {i}\widetilde{X}_{4}&{}\ \ \widetilde{X}_{1}-\mathbf {i}\widetilde{X}_{2}\\ \ \ \ \ \ \vdots &{}\ \ \ \ \ \ \ \vdots \\ \widetilde{X}_{4l+1}+\mathbf {i}\widetilde{X}_{4l+2}&{} -\widetilde{X}_{4l+3}-\mathbf {i}\widetilde{X}_{4l+4}\\ \widetilde{X}_{4l+3}-\mathbf {i}\widetilde{X}_{4l+4}&{} \ \ \widetilde{X}_{4l+1}-\mathbf {i}\widetilde{X}_{4l+2}\\ \ \ \ \ \ \vdots &{}\ \ \ \ \ \ \ \vdots \end{array}\right) , \end{aligned}$$
(2.43)

where \(A=0,1,\ldots ,2n-1,\)\(A'=0',1'.\) They satisfy the following commutating relations:

$$\begin{aligned} \begin{aligned}&\left[ \widetilde{Z}_{(2l)0'},\widetilde{Z}_{(2l+1)0'}\right] =8\left( \partial _{t_2}-\mathbf {i}\partial _{t_3}\right) ,\\&\left[ \widetilde{Z}_{(2l)1'},\widetilde{Z}_{(2l+1)1'}\right] =8\left( \partial _{t_2}+\mathbf {i}\partial _{t_3}\right) ,\\&\left[ \widetilde{Z}_{(2l)0'},\widetilde{Z}_{(2l+1)1'}\right] =\left[ \widetilde{Z}_{(2l)1'},\widetilde{Z}_{(2l+1)0'}\right] =-8\mathbf {i}\partial _{t_1}, \end{aligned} \end{aligned}$$
(2.44)

\(l=0,\ldots ,n-1,\) and any other bracket vanishes. So by embedding the real Lie algebra of \(\widetilde{{\mathscr {H}}}\) into the complex Lie algebra \(\mathfrak {u}_-\) by \(\widetilde{Z}_{(2l)A'}\mapsto Y_{lA'},\widetilde{Z}_{(2l+1)A'}\mapsto Y_{(n+l)A'}\) we get tangential k-Cauchy–Fueter complexes on \(\widetilde{{\mathscr {H}}}\) (cf. [37, Theorem 1.0.1]), on which \(G=\mathrm{Sp}(2(n+2),\mathbb {C})\) acts naturally.

Now, consider complexes on the right quaternionic Heisenberg group. We can show the following proposition as [38, Proposition 3.1].

Proposition 2.2

Under the transformation \(M_{\mathbf {a}}:{\mathbb {H}}^n\rightarrow {\mathbb {H}}^n,\)\(q\mapsto q'=q{\mathbf {a}}\) with \(\mathbf {a}=(a_{jk})\in GL(n,\mathbb {H}),\) where \(q=(q_1,q_2,\ldots ,q_{n})\) with \(q_{l+1}={x_{4l+1}}+\mathbf {i}{x_{4l+2}} +{\mathbf {j}}{x_{4l+3}}+{\mathbf {k}}{x_{4l+4}},\) we have

$$\begin{aligned} \overline{\partial }_{q_l}\left[ f(q{\mathbf {a}})\right] =\sum _{m=1}^n\left[ \overline{\partial }_{q'_m} (\bar{\mathbf {a}}_{l m} f) \right] (q{\mathbf {a}}), \end{aligned}$$
(2.45)

where \(\overline{\partial }_{q_{l+1}}=\partial _{x_{4l+1}}+\mathbf {i}\partial _{x_{4l+2}} +\mathbf {j}\partial _{x_{4l+3}}+\mathbf {k}\partial _{x_{4l+4}}.\)

Proof

Denote \({\widehat{q}}=(x_1,\ldots ,x_{4n}) \). Since \(M_{\mathbf {a}}\) defines a real linear transformation on the underlying vector space \(\mathbb {R}^{4n}\), we have \(\widehat{q{\mathbf {a}}}={\widehat{q}}{\mathbf {a}}^\mathbb {R} \) for some \((4n)\times (4n)\) real matrix \({\mathbf {a}}^\mathbb {R}\) associated with \({\mathbf {a}}.\) As the bth element of \(\widehat{q{\mathbf {a}}}\) is \(\sum _{a=1}^{4n}x_a{\mathbf {a}}^\mathbb {R}_{a b},\) we have

$$\begin{aligned} \frac{\partial }{\partial x_a}\left[ f(q{\mathbf {a}})\right] =\sum _{b=1}^{4n}\frac{\partial f}{\partial x_b}(q{\mathbf {a}}) {\mathbf {a}}^\mathbb {R}_{a b}. \end{aligned}$$

Note that we can write \(q_{l+1} = \sum _{j=1}^4{\mathbf {i}}_{j-1} x_{4l+j} \). Therefore,

$$\begin{aligned} \begin{aligned} M_{\mathbf {a}*}\overline{\partial }_{q_{l+1}}=&\sum _{j=1}^4{\mathbf {i}}_{j-1}M_{\mathbf {a}*}\frac{\partial }{\partial x_{4l +j}}=\sum _{b=1}^{4n}\sum _{j=1}^4{\mathbf {i}}_{j-1}\frac{\partial }{\partial x_{b}}{\mathbf {a}}^\mathbb {R}_{(4l +j)b}\\=&\sum _{b=1}^{4n}\sum _{j=1}^4{\mathbf {i}}_{j-1}\frac{\partial }{\partial x_{b}}\left( {{\mathbf {a}}}^\mathbb {R}\right) ^t_{b(4l +j)}=\sum _{m=1}^{ n} \overline{\partial }_{q'_m}\cdot \bar{\mathbf {a}}_{(l+1) m}, \end{aligned} \end{aligned}$$

by \(\left( {{\mathbf {a}}}^\mathbb {R}\right) ^t=\left( \overline{{{\mathbf {a}}}}^t\right) ^\mathbb {R}\), which can be proved as [38, Lemma 2.1 (1)]. The proposition is proved. \(\square \)

Corollary 2.2

Let \(\overline{Q}_{l+1}:=X_{4l+1}+\mathbf iX_{4l+2}+\mathbf jX_{4l+3}+\mathbf kX_{4l+4}.\) Then, \({R_\mathbf {a}}_*\left( \overline{Q}_1,\ldots ,\overline{Q}_n\right) =\left( \overline{Q}_1,\ldots ,\overline{Q}_n\right) \bar{{\mathbf {a}}}^t,\) for rotation \({R_\mathbf {a}}\) in (2.34) with \({\mathbf {a}}\in \mathrm{Sp}(n) \).

Since \(\overline{Q}_l=\overline{\partial }_{q_l}\) at the origin of \({\mathscr {H}},\) the above identity holds at the origin by Proposition 2.2. It holds at other places by the left invariance. By applying the representation \(\tau \) in (2.13), i.e., \(\tau (q_1q_2)=\tau (q_1)\tau (q_2)\) for any \(q_1,q_2\in \mathbb {H} \) (cf. [33, Proposition 2.1]), we get

$$\begin{aligned} \begin{aligned}&{R_\mathbf {a}}_*\left( \begin{array}{llllll}Z_{00'}&{}Z_{01'}&{}\cdots &{} Z_{(2l)0'}&{}Z_{(2l)1'}&{}\cdots \\ Z_{10'}&{}Z_{11'}&{}\cdots &{} Z_{(2l+1)0'}&{}Z_{(2l+1)1'}&{}\cdots \end{array}\right) \\&\quad =\left( \begin{array}{llllll}Z_{00'}&{}Z_{01'}&{}\cdots &{} Z_{(2l)0'}&{}Z_{(2l)1'}&{}\cdots \\ Z_{10'}&{}Z_{11'}&{}\cdots &{} Z_{(2l+1)0'}&{}Z_{(2l+1)1'}&{}\cdots \end{array}\right) \tau (\bar{{\mathbf {a}}}^t), \end{aligned} \end{aligned}$$
(2.46)

where \(\tau (\bar{{\mathbf {a}}}^t)\) is a \((2n)\times (2n)\) complex matrix with \(\overline{\mathbf {a}}_{jk}\) replaced by the \(2\times 2\) matrix \(\tau (\overline{\mathbf {a}}_{jk}).\) (2.46) implies that each column in (2.14) is not preserved under rotations (2.34) of \(\mathrm{Sp}(n).\) The commutativity (2.19) of each column that plays a very important role in the construction of our complexes (1.4) is destroyed. So by definition (2.23), (2.26) and (2.27), the differential operators \({\mathscr {D}}_j\)’s in the complex (1.4) in terms of \(Z_{A}^{A'}\)’s are not invariant under \(\mathrm{Sp}(n).\) Therefore, they are not invariant under \(\mathrm{Sp}(2(n+2),\mathbb {C}).\)

Another difference is that the kernel of the tangential k-Cauchy–Fueter in space of \(L^2\) integrable function on the left quaternionic Heisenberg group \(\widetilde{{\mathscr {H}}}\) is infinite dimensional [32], while it is trivial on the right quaternionic Heisenberg group \( {{\mathscr {H}}}\) , since such a function satisfies \(\Delta _b f=0\) by Proposition 2.4 and \(\ker \Delta _b=\{0\}\) in the \(L^2\) space.

On the other hand, if we choose the complex horizontal fields on \({\mathscr {H}}\)

$$\begin{aligned} \begin{aligned} \left( \widehat{Z}_{AA'}\right) :=\left( \begin{array}{c@{\quad }c} -\,Y_{1}+\mathbf {i}Y_{2}&{} -\,Y_{3}-\mathbf {i}Y_{4}\\ Y_{3}-\mathbf {i}Y_{4}&{} -\,Y_{1}-\mathbf {i}Y_{2}\\ \vdots &{}\vdots \\ -\,Y_{4l+1}+\mathbf {i}Y_{4l+2}&{}-\,Y_{4l+3}-\mathbf {i}Y_{4l+4}\\ Y_{4l+3}-\mathbf {i}Y_{4l+4}&{} -\,Y_{4l+1}-\mathbf {i}Y_{4l+2}\\ \vdots &{} \vdots \end{array}\right) \end{aligned} \end{aligned}$$
(2.47)

with \(Y_{4l+1}\) replaced by \(-Y_{4l+1},\) then \(\widehat{Z}_{AA'}\)’s satisfy

$$\begin{aligned} \begin{aligned}&\left[ \widehat{Z}_{(2l)0'},\widehat{Z}_{(2l+1)0'}\right] =8\left( \partial _{t_2}-\mathbf {i}\partial _{t_3}\right) ,\\&\left[ \widehat{Z}_{(2l)1'},\widehat{Z}_{(2l+1)1'}\right] =8\left( \partial _{t_2}+\mathbf {i}\partial _{t_3}\right) ,\\&\left[ \widehat{Z}_{(2l)0'},\widehat{Z}_{(2l+1)1'}\right] =\left[ \widehat{Z}_{(2l)1'},\widehat{Z}_{(2l+1)0'}\right] =-8\mathbf {i}\partial _{t_1}, \end{aligned} \end{aligned}$$

\(l=0,\ldots ,n-1,\) and any other bracket vanishes, i.e., we can embed the real Lie algebra of \({\mathscr {H}}\) into the complex Lie algebra \({{\mathfrak {u}}}_-.\) Then, the complexes (2.39) on \(\mathrm{U}_-\) induce a family of complexes on \(\mathscr {H}\) invariant under \(\mathrm{Sp}(2(n+2),\mathbb {C}).\) But the first operator is different from the first one in (1.4). Moreover, the \((n-1)\)th operator in the complex induced from \(\mathrm{U}_-\) is a linear combination of \(T_\beta \)’s (cf. [37, Proposition 4.3.3]), while the \((n-1)\)th operator in (1.4) involves only \(Z_{AA'}\)’s. So we get two different families of complexes on \( {{\mathscr {H}}}\). Here, changing \(Y_{4l+1}\) to \(-Y_{4l+1} \) corresponds to changing the sign before \(x_{4l+1}^2\) in the defining function (1.2) of the hypersurface \(\mathcal {S}\). The resulting hypersurface is essentially the boundary of the quaternionic Siegel domain.

On other quadratic hypersurface, there is no reason to expect that the restriction of the k-Cauchy–Fueter operators and complexes is invariant in general under the action of \(\mathrm{Sp}(n).\)

2.6 The adjoint operator

On a domain \(\Omega \subset \mathscr {H},\) denote the inner product

$$\begin{aligned} (u,v):=\int _{\Omega }u\cdot \overline{v}\hbox {d}V, \end{aligned}$$

for \(u,v\in L^2(\Omega ,\mathbb {C}),\) where \(\hbox {d}V\) is the Lebesgue measure on \(\mathscr {H}.\) The inner product of \(L^2(\Omega ,\mathscr {V}_1)\) is defined as

$$\begin{aligned} \langle f,h\rangle :=\sum _{A=0}^{2n-1} \sum _{A_2',\ldots ,A_k'=0',1'}\left( f_{AA_2'\ldots A_k'},h_{AA_2'\ldots A_k'}\right) \end{aligned}$$

for \(f,h\in L^2(\Omega ,\mathscr {V}_1),\) and \(\Vert f\Vert :=\langle f,f\rangle ^{\frac{1}{2}}.\) We define inner products of \(L^2(\Omega ,\mathscr {V}_0)\) and \(L^2(\Omega ,\mathscr {V}_2)\) similarly. Define the \(L^2\)-norm on \(\mathscr {H}/\mathscr {H}_{\mathbb {Z}}\) by

$$\begin{aligned} \Vert f\Vert ^2_{L^2(\mathscr {H}/\mathscr {H}_{\mathbb {Z}})}= \Vert f\Vert ^2_{L^2(\mathscr {F})}=\int _{\mathscr {F}}|f|^2\hbox {d}V. \end{aligned}$$

Proposition 2.3

The formal adjoint operator of \(Z_A^{A'}\) is

$$\begin{aligned} \left( Z_A^{A'}\right) ^*=\delta _{A'}^A,\quad \mathrm{where}\quad \delta ^A_{A'}:=-\overline{Z_A^{A'}}. \end{aligned}$$
(2.48)

Proof

For \(u,v\in C_0^\infty (\mathscr {H},\mathbb {C}),\) we have

$$\begin{aligned} (Y_au,v)=(u,-Y_av) \end{aligned}$$

by integration by part. So \(((Y_a\pm \mathbf {i}Y_b)u,v)=(u,-(Y_a\mp \mathbf {i}Y_b)v).\) Then, (2.48) holds since \(Z_A^{A'}\) has the form \(Y_a\pm \mathbf {i}Y_b\) for some a and b by (2.16). Thus, we have

$$\begin{aligned} \left( Z_A^{A'}u,v\right) =\left( u,\delta ^A_{A'}v\right) \end{aligned}$$
(2.49)

over \(\mathscr {H}.\) For (2.49) over \(\mathscr {H}/\mathscr {H}_{\mathbb {Z}},\) by using the unit partition, it is sufficient to show it for \(v\in C_0^\infty (\mathscr {H},\mathbb {C}).\) This case follows from the result over \(\mathscr {H}.\)\(\square \)

Lemma 2.3

For \(f\in C_0^1(\mathscr {H},\mathscr {V}_1)\) or \(C^1(\mathscr {H}/\mathscr {H}_{\mathbb {Z}},\mathscr {V}_1),\) we have

$$\begin{aligned} \left( \mathscr {D}_0^*f\right) _{A_1'\ldots A_k'}=\sum _{A=0}^{2n-1}\delta ^A_{(A_1'}f_{A_2'\ldots A_k')A}. \end{aligned}$$
(2.50)

Proof

The proof is similar to that for the k-Cauchy–Fueter operator over \(\mathbb {H}^n\) (cf. [40, Lemma 3.1]). For any \(g\in C^1(\mathscr {H}/\mathscr {H}_{\mathbb {Z}},\mathscr {V}_0),\) we have

$$\begin{aligned} \begin{aligned} \langle \mathscr {D}_0g,f\rangle =&\sum _{A,A_2',\ldots ,A_k'}\left( \sum _{A_1'} Z_A^{A_1'}g_{A_1'\ldots A_k'},f_{A_2'\ldots A_k'A}\right) =\sum _{A,A_1',\ldots ,A_k'}\left( g_{A_1'\ldots A_k'},\delta _{A_1'}^Af_{A_2'\ldots A_k'A}\right) \\=&\sum _{A_1',\ldots ,A_k'}\left( g_{A_1'\ldots A_k'},\sum _A\delta ^A_{(A_1'}f_{A_2'\ldots A_k')A}\right) =\langle g,\mathscr {D}_0^*f\rangle \end{aligned} \end{aligned}$$

by using (2.49) and symmetrization

$$\begin{aligned} \sum _{A_1',\ldots ,A_k'}\left( g_{A_1'\ldots A_k'},G_{A_1'\ldots A_k'}\right) =\sum _{A_1',\ldots ,A_k'}\left( g_{A_1'\ldots A_k'},G_{(A_1'\ldots A_k')}\right) \end{aligned}$$

for any \(g\in L^2(\mathscr {H},\odot ^k\mathbb {C}^2),G\in L^2(\mathscr {H},\otimes ^k\mathbb {C}^2).\) (cf. [40, (3.4)]). Here, we have to symmetrise the primed indices in \(\sum _A\delta ^A_{A_1'}f_{A_2'\ldots A_k'A}\) since only after symmetrization it becomes an element of \(C_0^1(\mathscr {H},\mathscr {V}_0).\)\(\square \)

\(\mathscr {D}_0^*\mathscr {D}_0\) is simple since it is diagonal by the following proposition.

Proposition 2.4

For \(f\in C^2(\Omega ,\mathscr {V}_0),\) we have

$$\begin{aligned} \mathscr {D}_0^*\mathscr {D}_0f=\Delta _bf. \end{aligned}$$

Proof

Recall that for a \(\otimes ^k\mathbb {C}^2\)-valued function \(F_{A_1'\ldots A_k'}\) symmetric in \(A_2'\ldots A_k',\) we have

$$\begin{aligned} F_{(A_1'\ldots A_k')}=\frac{1}{k}\left( F_{A_1'A_2'\ldots A_k'}+\cdots +F_{A_s'A_1'\ldots \widehat{A_s'}\ldots A_k'}+\cdots +F_{A_k'A_1'\ldots \widehat{A_k'}}\right) , \end{aligned}$$
(2.51)

by the definition of symmetrization (2.17). As usual, a hat means omittance of the corresponding index. Then, for fixed \(A'_1,\ldots ,A'_k=0',1',\)

$$\begin{aligned} \begin{aligned} \left( \mathscr {D}_0^*\mathscr {D}_0f\right) _{A_1'\ldots A_k'}=&\sum _A\delta ^A_{(A_1'} \left( \mathscr {D}_0f\right) _{A_2'\ldots A_k')A}=\frac{1}{k}\sum _{s=1}^k \delta ^A_{A_s'}\left( \mathscr {D}_0f\right) _{\ldots \widehat{A_s'}\ldots A_k'A}\\ =&-\frac{1}{k}\sum _{s=1}^k\sum _{A,A'}\overline{Z_A^{A'_{s}}} Z_A^{A'}f_{A'\ldots \widehat{A_s'}\ldots A_k'}\\=&\frac{1}{k}\sum _{s=1}^k\sum _{A'}\Delta _bf_{A' \ldots \widehat{A_s'}\ldots A_k'}\delta _{A_s'A'}=\Delta _bf_{A_1'\ldots A_k'}, \end{aligned} \end{aligned}$$
(2.52)

by using the following Lemma 2.4 and f symmetric in the primed indices, where \(\mathscr {D}_0^*\) is given by (2.50). The proposition is proved. \(\square \)

Lemma 2.4

For \(A',B'=0',1',\) we have

$$\begin{aligned} \sum _{A=0}^{2n-1}\overline{Z_A^{A'}}Z_A^{B'}=-\,\delta _{A'B'}\Delta _b. \end{aligned}$$
(2.53)

Proof

Note that

$$\begin{aligned} \begin{aligned} \overline{Z_{2l}^{0'}}Z_{2l}^{0'}+ \overline{Z_{2l+1}^{0'}}Z_{2l+1}^{0'}&= (-\,Y_{4l+3}+\mathbf {i}Y_{4l+4})(-Y_{4l+3}-\mathbf {i}Y_{4l+4})\\&\quad +\, (Y_{4l+1}+\mathbf {i}Y_{4l+2})(Y_{4l+1}-\mathbf {i}Y_{4l+2})\\ {}&=\sum _{k=1}^4Y_{4l+k}^2+\mathbf {i}[Y_{4l+3},Y_{4l+4}] -\mathbf {i}[Y_{4l+1},Y_{4l+2}]=\sum _{k=1}^4Y_{4l+k}^2, \end{aligned} \end{aligned}$$

by (2.21), whose summation over l gives us (2.53) for \(A'=B'=0'.\) Similarly, we have

$$\begin{aligned} \begin{aligned} \overline{Z_{2l}^{0'}}Z_{2l}^{1'}+ \overline{Z_{2l+1}^{0'}}Z_{2l+1}^{1'}&= (-Y_{4l+3}+\mathbf {i}Y_{4l+4})(-Y_{4l+1}-\mathbf {i}Y_{4l+2})\\&\quad +\, (Y_{4l+1}+\mathbf {i}Y_{4l+2})(-Y_{4l+3}+\mathbf {i}Y_{4l+4})\\ {}&=-\,[Y_{4l+1},Y_{4l+3}]-[Y_{4l+2},Y_{4l+4}]\\&\quad +\mathbf {i}[Y_{4l+1},Y_{4l+4}]-\mathbf {i}[Y_{4l+2},Y_{4l+3}]=0, \end{aligned} \end{aligned}$$

by (2.21), whose summation over l gives us (2.53) for \(A'=0,B'=1'.\) Similarly, (2.53) holds for \(A'=1,B'=0'\) and \(A'=B'=1'\) by

$$\begin{aligned} \begin{aligned} \overline{Z_{2l}^{1'}}Z_{2l}^{0'}+ \overline{Z_{2l+1}^{1'}}Z_{2l+1}^{0'}&=[Y_{4l+1},Y_{4l+3}]+[Y_{4l+2},Y_{4l+4}]\\&\quad +\,\mathbf {i}[Y_{4l+1},Y_{4l+4}]-\mathbf {i}[Y_{4l+2},Y_{4l+3}]= 0,\\ \overline{Z_{2l}^{1'}}Z_{2l}^{1'}+ \overline{Z_{2l+1}^{1'}}Z_{2l+1}^{1'}&=\sum _{k=1}^4Y_{4l+k}^2+\mathbf {i}[Y_{4l+1},Y_{4l+2}] -\mathbf {i}[Y_{4l+3},Y_{4l+4}]=\sum _{k=1}^4Y_{4l+k}^2. \end{aligned} \end{aligned}$$

Then, (2.53) follows. \(\square \)

3 The \(L^2\) estimate

We begin with the following Poincaré-type inequality, which was proved for general vector fields satisfying Hörmander’s condition (cf. [20, Theorem 2.1]). So it holds over \(\mathscr {H}.\)

Proposition 3.1

(Poincaré-type inequality) For each f with \(\sum _{a=1}^{4n}|Y_af|^2\in L^1(\mathscr {H}),\) we have

$$\begin{aligned} \int _{B_r}|f-f_{B_r}|^2 \mathrm{d}V\le Cr^2\int _{B_r}\sum _{a=1}^{4n}|Y_af|^2 \mathrm{d}V, \end{aligned}$$
(3.1)

where \(B_r\) is a ball of radius r and \(f_{B_r}={\int _{B_r}f\mathrm{d}V}/{\int _{B_r}\mathrm{d}V}.\)

We say \(f\in L^2(\mathscr {H}/\mathscr {H}_{\mathbb {Z}},\mathscr {V}_1)\) satisfies \(f\perp {\,constant}\)vectors if \(\langle f,C\rangle =0\) for any constant vector \(C\in \mathscr {V}_1.\)

Lemma 3.1

There exists some \(c>0\) such that

$$\begin{aligned} \left\langle \Delta _b f,f\right\rangle \ge c\Vert f\Vert ^2_{L^2\left( \mathscr {H}/\mathscr {H}_{\mathbb {Z}}\right) }, \end{aligned}$$

for \(f\in C^2\left( \mathscr {H}/\mathscr {H}_{\mathbb {Z}},\mathscr {V}_1\right) \) and \(f\perp \,\)constant vectors.

Proof

As \(\bigcup \limits _{(n,m)\in \mathscr {H}_{\mathbb {Z}}} \tau _{(n,m)}\mathscr {F}=\mathscr {H}\) by Proposition 2.1, we can choose some \(r>0\) and a finite number of elements \((n_i,m_i)\in \mathscr {H}_\mathbb {Z},i=1,\ldots ,N,\) such that

$$\begin{aligned} \mathscr {F}\subset B_r\subset \bigcup \limits _{i=1}^N\tau _{(n_i,m_i)}\mathscr {F}. \end{aligned}$$

Recall that if we identify \(f\in C^2(\mathscr {H}/\mathscr {H}_{\mathbb {Z}},\mathscr {V}_1)\) with a periodic function on \(\mathscr {H},\) so is \(Y_af.\) Then, the Poincaré-type inequality (3.1) implies that

$$\begin{aligned} \begin{aligned} N\sum _{a=1}^{4n}\Vert Y_a f\Vert ^2_{L^2\left( \mathscr {F}\right) }\ge \sum _{a=1}^{4n}\Vert Y_a {f}\Vert ^2_{L^2\left( B_r\right) }\ge \frac{1}{Cr^2}\int _{B_r}|{f}-f_{B_r}|^2\hbox {d}V\ge \frac{1}{Cr^2} \Vert f-f_{B_r}\Vert ^2_{L^2\left( \mathscr {F}\right) }. \end{aligned} \end{aligned}$$

Since \(f\bot \) constant vectors, we have

$$\begin{aligned} \Vert f-f_{B_r}\Vert ^2_{L^2(\mathscr {H}/\mathscr {H}_{\mathbb {Z}})} =\Vert f\Vert ^2_{L^2(\mathscr {H}/\mathscr {H}_{\mathbb {Z}})} +\Vert f_{B_r}\Vert ^2_{L^2(\mathscr {H}/\mathscr {H}_{\mathbb {Z}})}\ge \Vert f\Vert ^2_{L^2(\mathscr {H}/\mathscr {H}_{\mathbb {Z}})}. \end{aligned}$$

Thus, we find that

$$\begin{aligned} \begin{aligned} \left\langle \Delta _b f,f\right\rangle \ge c \Vert f-f_{B_r}\Vert ^2_{L^2(\mathscr {H}/\mathscr {H}_{\mathbb {Z}})}\ge c\Vert f\Vert ^2_{L^2(\mathscr {H}/\mathscr {H}_{\mathbb {Z}})}, \end{aligned} \end{aligned}$$

for constant \(c=\frac{1}{NCr^2}.\)\(\square \)

Lemma 3.2

(cf. [40, Lemma 2.1]) For any \(h,H\in \mathbb {C}^{2n}\otimes \mathbb {C}^{2n},\) we have

$$\begin{aligned} \sum _{A,B}h_{BA}\overline{H_{AB}}=\sum _{A,B}h_{AB}\overline{H_{AB}}- 2\sum _{A,B}h_{[AB]}\overline{H_{[AB]}}. \end{aligned}$$

We have the following \(L^2\) estimate.

Theorem 3.1

For \(n>3,k\ge 2,\) there exists some \(c_{n,k}>0\) such that

$$\begin{aligned} \Vert \mathscr {D}_{0}^*f\Vert ^2+\Vert \mathscr {D}_{1}f\Vert ^2\ge c_{n,k}\Vert f\Vert ^2, \end{aligned}$$
(3.2)

for \(f\in Dom(\mathscr {D}_1)\cap Dom(\mathscr {D}_0^*)\) and \(f\perp \,\)constant vectors over \(\mathscr {H}/\mathscr {H}_{\mathbb {Z}}\).

Proof

We use the \(L^2\) method for the k-Cauchy–Fueter operator on \(\mathbb {H}^n\) in [40]. Since \(C^2\) functions are dense in \( Dom(\mathscr {D}_1)\cap Dom(\mathscr {D}_0^*)\) for the compact manifold \(\mathscr {H}/\mathscr {H}_{\mathbb {Z}}\), it is sufficient to prove (3.2) for \(f\in C^2(\mathscr {H}/\mathscr {H}_{\mathbb {Z}},\odot ^{k-1}\mathbb {C}^2\otimes \mathbb {C}^{2n}).\) We have

$$\begin{aligned} \begin{aligned} k\langle \mathscr {D}_{0}^*f,\mathscr {D}_{0}^*f\rangle&= k\langle \mathscr {D}_{0}\mathscr {D}_{0}^*f,f\rangle =k\sum _{B,A_2',\ldots ,A_k'} \left( \sum _{A_1'}Z_B^{A_1'}\sum _{A}\delta ^A_{(A_1'}f_{A_2'\ldots A_k')A}, f_{A_2'\ldots A_k'B}\right) \\&=\quad \sum _{A,B,A_1',\ldots ,A_k'}\left( Z_B^{A_1'} \delta _{A_1'}^Af_{A_2' \ldots A_k'A}, f_{A_2' \ldots A_k'B}\right) \\ {}&\quad +\, \sum _{A,B,A_1',\ldots ,A_k'}\sum _{s=2}^{k}\left( Z_B^{A_1'} \delta _{A_s'}^Af_{A_1'\ldots \widehat{A_s'}\ldots A_k'A},f_{A_2' \ldots A_k'B}\right) =:\Sigma _0+\Sigma _1, \end{aligned} \end{aligned}$$
(3.3)

by using (2.51) to expand the symmetrization. Note that

$$\begin{aligned} \begin{aligned} \Sigma _0=\sum _{A_1',\ldots ,A_k'}\left( \sum _A\delta _{A_1'}^A f_{A_2' \ldots A_k'A}, \sum _B\delta _{A_1'}^Bf_{A_2' \ldots A_k'B}\right) =\sum _{A_1',\ldots ,A_k'} \left\| \sum _A\delta _{A_1'}^Af_{A_2' \ldots A_k'A}\right\| ^2\ge 0, \end{aligned}\nonumber \\ \end{aligned}$$
(3.4)

and

$$\begin{aligned} \begin{aligned} \Sigma _1&=\quad \sum _{s=2}^{k}\sum _{A,B,A_1',\ldots ,A_k'} \left( \delta _{A_s'}^AZ_B^{A_1'} f_{A_1'\ldots \widehat{A_s'}\ldots A_k'A},f_{A_2' \ldots A_k'B}\right) \\&\quad +\,\sum _{s=2}^{k}\sum _{A,B,A_1',\ldots ,A_k'} \left( \left[ Z_B^{A_1'},\delta _{A_s'}^A \right] f_{A_1'\ldots \widehat{A_s'}\ldots A_k'A},f_{A_2' \ldots A_k'B}\right) =: \Sigma _{11}+{\mathscr {C}} \end{aligned} \end{aligned}$$
(3.5)

by using commutators. For the first sum, we have

$$\begin{aligned} \begin{aligned} \Sigma _{11}&=\sum _{s=2}^{k}\sum _{A,B,A_1',\ldots ,A_k'} \left( Z_B^{A_1'}f_{A_1'\ldots \widehat{A_s'}\ldots A_k'A},Z_A^{A_s'}f_{A_2' \ldots A_k'B} \right) \\ {}&=\sum _{s=2}^{k}\sum _{A,B}\sum _{\widehat{A_1'},\ldots , \widehat{A_s'},\ldots ,A_k'} \left( \sum _{A_1'}Z_B^{A_1'}f_{A_1'\ldots \widehat{A_s'}\ldots A_k'A},\sum _{A_s'}Z_A^{A_s'}f_{A_s'A_2'\ldots \widehat{A_s'}\ldots A_k'B} \right) \\ {}&=(k-1)\sum _{B_3',\ldots ,B_k'=0',1'}\sum _{A,B} \left( \sum _{A'}Z_B^{A'}f_{A A'B_3'\ldots B_k'},\sum _{A'}Z_A^{A'}f_{B A'B_3'\ldots B_k'}\right) \end{aligned} \end{aligned}$$
(3.6)

by relabeling indices and f symmetric in the primed indices. Then, by applying Lemma 3.2 with \(h_{BA}=\sum _{A'}Z_B^{A'}f_{AA'B_3'\ldots B_k'}\) and \(H_{A B}=\sum _{A'}Z_A^{A'}f_{BA'B_3'\ldots B_k'}\) for fixed \(B_3',\ldots ,B_k',\) we get

$$\begin{aligned} \begin{aligned} \Sigma _{11}&=(k-1)\sum _{B_3',\ldots ,B_k'}\sum _{A,B}\left( \left\| \sum _{A'}Z_{A}^{A'}f_{B A'B_3'\ldots B_k'}\right\| ^2-2\left\| \sum _{A'}Z_{[A}^{A'}f_{B]A'B_3'\ldots B_k'}\right\| ^2\right) \\&=(k-1)\sum _{B_3',\ldots ,B_k'}\sum _{A,B} \left\| \sum _{A'}Z_{A}^{A'}f_{B A'B_3'\ldots B_k'}\right\| ^2-\frac{k-1}{2}\Vert \mathscr {D}_{1}f\Vert ^2, \end{aligned} \end{aligned}$$
(3.7)

where

$$\begin{aligned} \begin{aligned}&\sum _A\left\| \sum _{A'}Z_{A}^{A'}f_{B A'B_3'\ldots B_k'}\right\| ^2=\sum _{A,A',B'}\left( Z_{A}^{A'}f_{B A'B_3'\ldots B_k'}, Z_{A}^{B'}f_{B B'B_3'\ldots B_k'}\right) \\&\quad =\sum _{A',B'}\left( -\sum _{A}\overline{Z_A^{B'}} Z_A^{A'}f_{B A'B_3'\ldots B_k'}, f_{B B'B_3'\ldots B_k'}\right) =\sum _{B'}\left( \Delta _b f_{BB'B_3'\ldots B_k'},f_{BB'B_3'\ldots B_k'}\right) \end{aligned} \end{aligned}$$
(3.8)

by Lemma 2.4. Thus, by substituting (3.4)–(3.5) and (3.7)–(3.8) to (3.3), we get

$$\begin{aligned} k\left\| \mathscr {D}^*_0f\right\| ^2+ \frac{k-1}{2}\left\| \mathscr {D}_1f\right\| ^2\ge (k-1)\langle \Delta _b f,f\rangle +{\mathscr {C}}. \end{aligned}$$
(3.9)

To control the commutator term \({\mathscr {C}}\) in (3.5), note that

$$\begin{aligned} \overline{Z_{2l}^{0'}}=Z_{2l+1}^{1'},\quad \overline{Z_{2l}^{1'}}=-Z_{2l+1}^{0'} \end{aligned}$$

by (2.16). Then, it follows from Lemma 2.1 that (1)

$$\begin{aligned} \left[ Z_A^{A'},\overline{Z_B^{B'}}\right] =0,\quad \mathrm{for}\ A'\ne B', \end{aligned}$$
(3.10)

\(A,B=0,\ldots ,2n-1;\) (2) for \(A'=B',\) we have

$$\begin{aligned} \begin{aligned} \left[ Z_{2l}^{0'},\overline{Z_{2l+1}^{0'}}\right]&= \left[ Z_{2l}^{1'},\overline{Z_{2l+1}^{1'}}\right] = -8\left( \partial _{s_2}+\mathbf {i}\partial _{s_3}\right) ,\\ {\left[ Z_{2l+1}^{0'},\overline{Z_{2l}^{0'}}\right] }&= {\left[ Z_{2l+1}^{1'},\overline{Z_{2l}^{1'}}\right] } =8\left( \partial _{s_2}-\mathbf {i}\partial _{s_3}\right) ,\\ \left[ Z_{2l}^{0'},\overline{Z_{2l}^{0'}}\right]&= \left[ Z_{2l}^{1'},\overline{Z_{2l}^{1'}}\right] =- \left[ Z_{2l+1}^{0'},\overline{Z_{2l+1}^{0'}}\right] =- \left[ Z_{2l+1}^{1'},\overline{Z_{2l+1}^{1'}}\right] =8\mathbf {i}\partial _{s_1}; \end{aligned} \end{aligned}$$
(3.11)

(3) if \(\{A,B\}\ne \{2l,2l+1\}\) for any l,  then \(\left[ Z_A^{A'},\overline{Z_B^{B'}}\right] =0\) for any \(A',B'.\) Thus, we have

$$\begin{aligned} \begin{aligned} {\mathscr {C}}&= \sum _{s=2}^k\sum _{A,B,A_1',\ldots ,A_k'}\left( \left[ Z_B^{A_1'}, -\overline{Z^{A_s'}_A}\right] f_{A'_1\ldots \widehat{A_s'}\ldots A_k'A},f_{A_2'\ldots A_k'B}\right) \\ {}&=-\,(k-1) \sum _{A,B,B',B_3',\ldots ,B_k'}\left( \left[ Z_B^{B'}, \overline{Z^{B'}_A}\right] f_{B'B_3'\ldots B_k'A},f_{B'B_3'\ldots B_k'B}\right) ,\\ {}&=-\,(k-1)\sum _{B',B_3',\ldots ,B_k'}\sum _{l=0}^{n-1} \left\{ \left( \left[ Z_{2l}^{B'}, \overline{Z_{2l}^{B'}}\right] f_{B'B_3'\ldots B_k'(2l)} ,f_{B'B_3'\ldots B_k'(2l)}\right) \right. \\&\quad +\,\left( \left[ Z_{2l}^{B'}, \overline{Z_{2l+1}^{B'}}\right] f_{B'B_3'\ldots B_k'(2l+1)} ,f_{B'B_3'\ldots B_k'(2l)}\right) \\&\quad +\,\left( \left[ Z_{2l+1}^{B'}, \overline{Z_{2l}^{B'}}\right] f_{B'B_3'\ldots B_k'(2l)} ,f_{B'B_3'\ldots B_k'(2l+1)}\right) \\&\quad +\,\left. \left( \left[ Z_{2l+1}^{B'}, \overline{Z_{2l+1}^{B'}}\right] f_{B'B_3'\ldots B_k'(2l+1)} ,f_{B'B_3'\ldots B_k'(2l+1)}\right) \right\} \end{aligned} \end{aligned}$$

by using (1) and (3) above, relabeling indices and f symmetric in the primed indices. Apply (3.11) to \({\mathscr {C}}\) above to get

$$\begin{aligned} \begin{aligned} {\mathscr {C}}&=-\,8(k-1) \sum _{B',B_3',\ldots ,B_k'} \left\{ \sum _{A=0}^{2n-1}(-1)^{A}\left( \mathbf {i}\partial _{s_1} f_{B'B_3'\ldots B_k'A},f_{B'B_3' \ldots B_k'A}\right) \right. \\&\quad +\,\sum _{l=0}^{n-1} \left( -( \partial _{s_2}+\mathbf {i}\partial _{s_3})f_{B'B_3'\ldots B_k'(2l+1)}, f_{B'B_3' \ldots B_k'(2l)}\right) \\&\quad \left. +\,\sum _{l=0}^{n-1} \left( (\partial _{s_2}-\mathbf {i}\partial _{s_3})f_{B'B_3' \ldots B_k'(2l)}, f_{B'B_3' \ldots B_k'(2l+1)}\right) \right\} . \end{aligned} \end{aligned}$$
(3.12)

For any \(u,v\in C^1({\mathscr {H}}/\mathscr {H}_\mathbb {Z},\mathbb {C}),\) we have

$$\begin{aligned} \begin{aligned} 8\left( \partial _{s_1}u,v\right) =-\frac{1}{n}\sum _{l=0}^{n-1}\left( [Y_{4l+1},Y_{4l+2}]u+[Y_{4l+3},Y_{4l+4}] u,v\right) , \end{aligned} \end{aligned}$$

by (2.21). As

$$\begin{aligned} \begin{aligned} \left| \left( [Y_{a},Y_{b}] u,v\right) \right|&=\left| \left( Y_{b} u,-{Y_{a}}v\right) +\left( Y_{a} u,{Y_{b}}v\right) \right| \\&\le \frac{1}{2}\left( \Vert Y_{a}u\Vert ^2+\Vert Y_{b}u\Vert ^2+\Vert Y_{a}v\Vert ^2+\Vert Y_{b}v\Vert ^2\right) , \end{aligned} \end{aligned}$$

for \(a,b=1,\ldots ,4n,\) we get

$$\begin{aligned} \begin{aligned} \left| 8\left( \partial _{s_1}u,v\right) \right| \le \frac{1}{2n}\sum _{a=1}^{4n}\left( \Vert Y_{a}u\Vert ^2+\Vert Y_{a}v\Vert ^2\right) . \end{aligned} \end{aligned}$$
(3.13)

Similarly, we have

$$\begin{aligned} \begin{aligned} \left| \left( 8(\partial _{s_2}\pm \mathbf {i}\partial _{s_3}) u,v\right) \right|&\le \frac{1}{n}\sum _{a=1}^{4n}\left( \Vert Y_{a}u\Vert ^2+ \Vert Y_{a}v\Vert ^2\right) . \end{aligned} \end{aligned}$$
(3.14)

Then, apply (3.13)–(3.14) to the right-hand side of (3.12) to get

$$\begin{aligned} |{\mathscr {C}}|\le (k-1)\frac{3}{n}\sum _{A,B',B_3',\ldots ,B_k'} \sum _{a=1}^{4n}\left\| Y_{a}f_{B'B_3'\ldots B_k'A}\right\| ^2= \frac{3(k-1)}{n}\left\langle \Delta _bf,f\right\rangle . \end{aligned}$$
(3.15)

So it follows from estimate (3.9) that

$$\begin{aligned} \begin{aligned} k\left\| \mathscr {D}^*_0f\right\| ^2+ \frac{k-1}{2}\left\| \mathscr {D}_1f\right\| ^2\ge (k-1)\left( 1-\frac{3}{n} \right) \left\langle \Delta _bf,f\right\rangle . \end{aligned} \end{aligned}$$
(3.16)

Now, by applying Lemma 3.1 we get (3.2). \(\square \)

4 Hartogs’ phenomenon

4.1 The nonhomogeneous tangential k-Cauchy–Fueter equation over \(\mathscr {H}/\mathscr {H}_{\mathbb {Z}}\)

Consider the Hilbert subspace \(\mathcal {L}\) consisting of \(f\in L^2\left( \mathscr {H}/\mathscr {H}_{\mathbb {Z}},\mathscr {V}_1\right) \) and \(f\perp \) constant vectors. The domain of \(\Box _1\) over \(\mathcal {L}\) is

$$\begin{aligned} \mathrm{Dom}(\Box _1):=\left\{ f\in \mathcal {L}:f\in \mathrm{Dom}(\mathscr {D}_0^*)\cap \mathrm{Dom}(\mathscr {D}_1),\mathscr {D}_0^*f\in \mathrm{Dom}(\mathscr {D}_0),\mathscr {D}_1f\in \mathrm{Dom}(\mathscr {D}_1^*)\right\} . \end{aligned}$$

Proposition 4.1

The associated Hodge–Laplacian \(\Box _1\) is densely defined, closed, self-adjoint and nonnegative operator on \(\mathcal {L}.\)

The proof is exactly the same as that of Proposition 3.1 in [40] since \(\mathcal {L}\oplus \{const.\}=L^2(\mathscr {H}/\mathscr {H}_\mathbb {Z}, \mathscr {V}_1)\), and the action of \(\Box _1\) on \(\{const.\}\) is trivial. We omit the detail. Now, we can find solution to (1.7)–(1.8), whose proof is similar to that of Theorem 1.2 in [40] for the k-Cauchy–Fueter operator on \(\mathbb {H}^n.\)

Theorem 4.1

Suppose that \(\mathrm{dim}\ \mathscr {H}\ge 19 \) and \(k=2,3,\ldots \). If \(f\in \mathrm{Dom}(\mathscr {D}_1)\) is \(\mathscr {D}_1\)-closed and \(f\perp \) constant vectors, then there exist \(u\in L^2\left( \mathscr {H}/\mathscr {H}_{\mathbb {Z}},\mathscr {V}_0\right) \) such that

$$\begin{aligned} \mathscr {D}_0u=f. \end{aligned}$$

Proof

The \(L^2\) estimate (3.2) implies

$$\begin{aligned} c_{n,k}\Vert g\Vert ^2\le \Vert \mathscr {D}_{0}^*g\Vert ^2+\Vert \mathscr {D}_{1}g\Vert ^2 =\langle \Box _1g,g \rangle \le \Vert \Box _1g\Vert \Vert g\Vert , \end{aligned}$$

for \(g\in \mathrm{Dom}(\Box _1),\) i.e.,

$$\begin{aligned} c_{n,k}\Vert g\Vert \le \Vert \Box _1g\Vert . \end{aligned}$$
(4.1)

Thus, \(\Box _1:\mathrm{Dom}(\Box _1)\rightarrow \mathcal {L}\) is injective. This together with the self-adjointness of \(\Box _1\) by Proposition 4.1 implies the density of the range. For fixed \(f\in \mathcal {L},\) the complex anti-linear functional

$$\begin{aligned} l_f: \Box _1g\longrightarrow \langle f,g\rangle \end{aligned}$$

is then well-defined on a dense subspace of \(\mathcal {L}.\) It is finite since

$$\begin{aligned} |l_f(\Box _{1}g)|=|\langle f,g\rangle |\le \Vert f\Vert \Vert g\Vert \le \frac{1}{c_{n,k}}\Vert f\Vert \Vert \Box _1g\Vert \end{aligned}$$

for any \(g\in \mathrm{Dom}(\Box _1),\) by (4.1). So \(l_f\) can be uniquely extended to a continuous anti-linear functional on \(\mathcal {L}.\) By the Riesz representation theorem, there exists a unique element \(h\in \mathcal {L}\) such that \(l_f(F)=\langle h,F\rangle \) for any \(F\in \mathcal {L},\) and \(\Vert h\Vert =\Vert l_f\Vert \le \frac{1}{c_{n,k}}\Vert f\Vert .\) Then, we have

$$\begin{aligned} \langle h,\Box _1g\rangle =\langle f,g\rangle \end{aligned}$$

for any \(g\in \mathrm{Dom}(\Box _1).\) This implies that \(h\in \mathrm{Dom}(\Box _1^*)\) and \(\Box _1^*h=f,\) and so \(h\in \mathrm{Dom}(\Box _1)\) and \(\Box _1h=f\) by self-adjointness of \(\Box _1.\) We write \(h=Nf.\) Then, \(\Vert Nf\Vert \le \frac{1}{c_{n,k}}\Vert f\Vert .\)

Since \(Nf\in \mathrm{Dom}(\Box _1),\) we have \(\mathscr {D}_0^*Nf\in \mathrm{Dom}(\mathscr {D}_0),\)\(\mathscr {D}_1Nf\in \mathrm{Dom}(\mathscr {D}_1^*),\) and

$$\begin{aligned} \mathscr {D}_0\mathscr {D}_0^*Nf=f-\mathscr {D}_1^*\mathscr {D}_1Nf \end{aligned}$$
(4.2)

by \(\Box _1Nf=f.\) Because f and \(\mathscr {D}_0F\) for any \(F\in \mathrm{Dom}(\mathscr {D}_0)\) are both \(\mathscr {D}_1\)-closed, the above identity implies \(\mathscr {D}_1^*\mathscr {D}_1Nf\in \mathrm{Dom}(\mathscr {D}_1)\) and so \(\mathscr {D}_1\mathscr {D}_1^*\mathscr {D}_1Nf=0.\) Then,

$$\begin{aligned} 0=\langle \mathscr {D}_1\mathscr {D}_1^*\mathscr {D}_1Nf,\mathscr {D}_1Nf \rangle =\Vert \mathscr {D}_1^*\mathscr {D}_1Nf\Vert ^2, \end{aligned}$$

i.e., \(\mathscr {D}_1^*\mathscr {D}_1Nf=0.\) Hence, \(\mathscr {D}_0\mathscr {D}_0^*Nf=f\) by (4.2). \(\square \)

4.2 Proof of Hartogs’ phenomenon

We need the analytic hypoellipticity of \(\Delta _b\). Let G be a nilpotent Lie group of step 2,  and its Lie algebra \(\mathfrak {g}\) has decomposition: \(\mathfrak {g}=\mathfrak {g}_1\oplus \mathfrak {g}_2\) satisfying \([\mathfrak {g}_1,\mathfrak {g}_1]\subset \mathfrak {g}_2,\ [\mathfrak {g},\mathfrak {g}_2]=0.\) Consider the condition (H): For any \(\lambda \in \mathfrak {g}_2^*{\setminus }\{0\},\) the antisymmetric bilinear form

$$\begin{aligned} B_\lambda (Y,Y')=\langle \lambda ,[Y,Y']\rangle , \end{aligned}$$

for \(Y,Y'\in \mathfrak {g}_1\) is nondegenerate. Métivier proved the following theorem for analytic hypoellipticity.

Theorem 4.2

([24, Theorem 0]) Let P be a homogeneous left invariant differential operator on a nilpotent Lie group G satisfies condition (H). Then, the following are equivalent:

  1. (i)

    P is analytic hypoelliptic;

  2. (ii)

    P is \(C^\infty \) hypoelliptic.

Corollary 4.1

\(\Delta _b\) is analytic hypoelliptic on a domain \(\Omega \subset \mathscr {H},\) i.e., for any distribution \(u\in S'(\Omega )\) such that \(\Delta _bu\) is analytic, u must be also analytic.

Proof

It follows from the well-known subellipticity of \(\Delta _b \) that u is locally \(C^{k+1}\) if \(\Delta _bu\) is locally \(C^{k }\). So \(\Delta _b \) is \(C^\infty \) hypoelliptic. To obtain the analytic hypoellipticity of \(\Delta _b \) by applying Theorem 4.2, it is sufficient to check the condition (H) for the right quaternionic Heisenberg group \(\mathscr {H}\). In this case, \(\mathfrak {g}_1=\mathrm{span}\{Y_1,\ldots ,Y_{4n}\},\)\(\mathfrak {g}_2=\mathrm{span}\left\{ \partial _{s_1}, \partial _{s_2}, \partial _{s_3}\right\} ,\) where \(Y_1,\ldots ,Y_{4n}\) is the left invariant vector fields in (2.4). Let \(\lambda \in \mathfrak {g}_2^*{\setminus }\{0\}.\) For \(Y_{4l+j},Y_{4l+j'}\in \mathfrak {g}_1,\) we have

$$\begin{aligned} B_\lambda (Y_{4l+j},Y_{4l'+j'})=\langle \lambda ,[Y_{4l+j},Y_{4l'+j'}]\rangle =4\delta _{ll'} \sum _{\beta =1}^{3}B_{jj'}^{\beta }\lambda (\partial _{s_{\beta }})=4\delta _{ll'} \sum _{\beta =1}^{3}B_{jj'}^{\beta }\lambda _{\beta }, \end{aligned}$$

by (2.5), if we write \(\lambda (\partial _{s_{\beta }})=\lambda _{\beta } \). Then, the matrix associated with \(B_\lambda \) is

$$\begin{aligned} 4\sum _{\beta =1}^{3}\left( \begin{array}{ccc}\lambda _{\beta }B^{\beta }&{}&{}\\ {} &{}\ddots &{}\\ &{}&{}\lambda _{\beta }B^{\beta }\end{array}\right) ,\quad \mathrm{where}\ \sum _{\beta =1}^{3}\lambda _{\beta }B^{\beta }=\left( \begin{array}{cccc}0&{}\quad -\,\lambda _1&{}\quad -\,\lambda _2&{}\quad -\,\lambda _3\\ \lambda _1&{}\quad 0&{}\quad -\,\lambda _3 &{}\quad \lambda _2\\ \lambda _2&{}\quad \lambda _3&{}\quad 0&{}\quad -\,\lambda _1\\ \lambda _3&{}\quad -\,\lambda _2 &{}\quad \lambda _1&{}\quad 0\end{array}\right) , \end{aligned}$$
(4.3)

whose determinant is \( \left( \lambda _1^2+\lambda _2^2+\lambda _3^2\right) ^{2n}\) by direct calculation. So \(B_\lambda \) is nondegenerate for \(\lambda \in \mathfrak {g}_2^*{\setminus }\{0\},\) i.e., \(\mathscr {H}\) satisfies condition (H). \(\square \)

Liouville-type theorems hold for SubLaplacian \(\Delta _b\) on the right quaternionic Heisenberg group by the following general theorem of Geller.

Theorem 4.3

([14, Theorem 2]) Let \(\mathscr {L}\) be a homogeneous hypoelliptic left invariant differential operator on a homogeneous group G. Suppose \(u\in S'(G)\) and \(\mathscr {L}u=0.\) Then, u is a polynomial.

Theorem 4.4

Let \(\widetilde{\Omega }\) be an open set in \(\mathscr {F}\) such that \(\widetilde{{\Omega }}\Subset \mathring{\mathscr {F}}\) and \(\mathscr {F}{\setminus }\widetilde{\Omega }\) are connected. If \(f\in C^1(\mathscr {H}/\mathscr {H}_{\mathbb {Z}},\mathscr {V}_1)\) with \(\mathrm{supp}f\subset \widetilde{\Omega }\) is \(\mathscr {D}_1\)-closed and \(f\perp \) constant vectors, then there exist \(u\in C^2\left( \mathscr {H}/\mathscr {H}_{\mathbb {Z}},\mathscr {V}_0\right) \) such that

$$\begin{aligned} \mathscr {D}_0u=f, \end{aligned}$$
(4.4)

with \(\mathrm{supp}\,u\subset \widetilde{\Omega }.\)

Proof

By Theorem 4.1, we can find a solution \(u\in L^2\left( \mathscr {H}/\mathscr {H}_{\mathbb {Z}},\mathscr {V}_0\right) \) to (4.4). For \(c\in \mathbb {{H}}\), denote

$$\begin{aligned} \mathscr {H}_c':=\{(q',c,s)\in \mathscr {H}:q'\in \mathbb {H}^{n-1},s\in \mathbb {R}^3\}. \end{aligned}$$

We see that \(\mathscr {H}'_c\cap \Omega =\emptyset \) for |c| small by \(\widetilde{\Omega }\Subset \mathring{\mathscr {F}}.\)

Since \(\mathscr {D}_0u=0\) on \(\left( \mathscr {H}/\mathscr {H}_{\mathbb {Z}}\right) {\setminus }\widetilde{\Omega },\) we have \(\mathscr {D}_0^*\mathscr {D}_0u=0,\), and then, by Proposition 2.4\(\Delta _bu_{A'_1\ldots A'_{k }A}=0\) on \(\left( \mathscr {H}/\mathscr {H}_{\mathbb {Z}}\right) {\setminus }\widetilde{\Omega }\) in the sense of distributions for any fixed \(A'_1,\ldots ,A'_{k },A\). So it is real analytic on \(\left( \mathscr {H}/\mathscr {H}_{\mathbb {Z}}\right) {\setminus }\widetilde{\Omega }\) by Corollary 4.1. Moreover, u is \(C^2\) on \(\mathscr {H}/\mathscr {H}_{\mathbb {Z}}\) by subellipticity of \(\Delta _b \). In particular, \(u(q',c,s)\) is well-defined on \(\mathscr {H}'_c/\mathscr {H}'_{\mathbb {Z}}\) as a real analytic function. So it can be extended to a periodic function over \(\mathscr {H}'_c\) by (2.11). Now, let \(\mathscr {D}_0'\) be the tangential k-Cauchy–Fueter operator on \(\mathscr {H}_c',\) i.e., \(\mathscr {D}_0'u\) is a \(\odot ^{k-1}\mathbb {C}^2\otimes \mathbb {C}^{2n-2}\)-valued function with

$$\begin{aligned} \left( \mathscr {D}_0' u\right) _{AA_2'\ldots A_{k}'}= (\mathscr {D}_0 u )_{AA_2'\ldots A_{k}'} , \quad A=0,1,\ldots ,2n-3. \end{aligned}$$

By applying Proposition 2.4 to \(\mathscr {H}_c',\) we see that \(\Delta _b'u=0,\) where \(\Delta _b'=-\sum _{a=0}^{4n-5}Y_a^2.\) Then, apply Liouville-type Theorem 4.3 to the group \(\mathscr {H}_c'\) and \(\Delta _b'\) to get

$$\begin{aligned} u(\cdot ,c,\cdot )= \mathrm{a\ polynomial}\ \mathrm{on}\ \mathscr {H}'_c, \end{aligned}$$

which must be a constant by periodicity. Thus, u only depends on the variable \(q_n.\)

Similarly, we can prove u is a constant on the subgroup

$$\begin{aligned} \mathscr {H}_0'':=\{(0,q_n,s)\in \mathscr {H};q_n\in \mathbb {H},s\in \mathbb {R}^3\}. \end{aligned}$$

Now, if replacing u by \(u-\) const., we see that u vanishes in a neighborhood of \(\mathscr {H}_0''.\) Consequently, by the identity theorem for real analytic functions it vanishes on the connected component \(\mathscr {F}{\setminus }\widetilde{{\Omega }}.\) Thus, \(\mathrm{supp}\,u\subset \widetilde{\Omega }.\)\(\square \)

The solution with supp\(\,u\subset \widetilde{\Omega }\) above plays the role of compactly supported solution to \(\overline{\partial }\) equation or the tangential k-Cauchy–Fueter equations (cf., e.g., [17, 35]). It leads to Hartogs’ extension phenomenon as follows.

The proof of Theorem 1.1

Without loss of generality, we can assume \(\Omega \Subset \mathring{\mathscr {F}}\) by dilating if necessary. Let \(\chi \in C_0^{\infty }(\Omega )\) be equal to 1 in a neighborhood of K such that \(\mathscr {F}{\setminus }\mathrm{supp}\,\chi \) is connected. Set

$$\begin{aligned} \widetilde{u}(\xi ):=\left\{ \begin{array}{ll}(1-\chi )u(\xi ),\quad &{}\quad \xi \in \Omega {\setminus } K\\ 0,\quad &{}\quad \xi \in K\end{array} \right. . \end{aligned}$$

Then, \(\widetilde{u}\in C^{\infty }(\Omega ),\) and \(\widetilde{u}|_{\Omega {\setminus } \mathrm{supp}\,{\chi }}=u|_{\Omega {\setminus } \mathrm{supp}\,\chi }.\) We have

$$\begin{aligned} \mathscr {D}_0\tilde{u}=\mathscr {D}_0 ((1-\chi ) {u})=:f \end{aligned}$$

on \(\mathscr {H},\) where \(f_{A_2'\ldots A_k'A}=-\sum _{A_1'}Z_A^{A_1'}\chi \cdot u_{A_1'\ldots A_k'}\) by \(\mathscr {D}_0u=0\) on \(\Omega {\setminus } K.\) Hence, \(f\in C_0^\infty (\mathscr {H},\mathscr {V}_1)\) vanishes in K and outside \(\Omega ,\) satisfying \(\mathscr {D}_1f=\mathscr {D}_1\mathscr {D}_0\tilde{u}=0\) by (2.30). We can extend f to a periodic function and view it as an element of \(C^\infty (\mathscr {H}/\mathscr {H}_\mathbb {Z},\mathscr {V}_1).\)

Denote

$$\begin{aligned} c:=\frac{\int _{\mathscr {H}/\mathscr {H}_\mathbb {Z}}f\hbox {d}V}{ \int _{\mathscr {H}/\mathscr {H}_\mathbb {Z}}\hbox {d}V}\in \mathscr {V}_1. \end{aligned}$$

Then, we have \((f-c) \perp \,\)constant vectors. It follows from Theorem 4.4 that there exists a solution \(\widetilde{U}\in C^2(\mathscr {H}/\mathscr {H}_\mathbb {Z},\mathscr {V}_0)\) to \(\mathscr {D}_0\widetilde{U}=f-c,\) which vanishes outside \(\widetilde{\Omega }:=\mathrm{supp}\,\chi \). Then, \(\mathscr {D}_0(\widetilde{u}-\widetilde{U})=c\) on \(\mathscr {H}/\mathscr {H}_\mathbb {Z}.\) So \(c=\mathscr {D}_0\widetilde{u}|_{\Omega {\setminus } \widetilde{\Omega }}=\mathscr {D}_0 {u}|_{\Omega {\setminus } \widetilde{\Omega }}=0.\) Therefore, \( U=\widetilde{u}-\widetilde{U} \) is k-CF in \(\Omega \) since \(\mathscr {D}_0(\widetilde{u}-\widetilde{U})=0.\) Note that \(\widetilde{U}\equiv 0\) outside \(\widetilde{\Omega }\) and \( \mathscr {F} {\setminus } \widetilde{\Omega }\) is connected. So \(U=u\) in \(\Omega {\setminus } \widetilde{\Omega }.\) Then, \(U=u\) in \(\Omega {\setminus } K\) by the identity theorem for real analytic functions. The theorem is proved. \(\square \)

5 The restriction of the k-Cauchy–Fueter operator to the hypersurface \(\mathcal {S}\)

5.1 The nilpotent Lie groups of step two associated with quadratic hypersurfaces

Let \((x_1,\ldots ,x_{4n},\)\(t_1,t_2,t_3)\) be coordinates of \(\mathbb {R}^{4n+3}.\) Now, consider general quadratic hypersurfaces \(\widehat{\mathcal {S}}\) defined by

$$\begin{aligned} \rho =\mathrm{Re}\, q_{n+1}-\phi (q'),\quad \mathrm{where}\quad \phi =\sum _{k=1}^{4n}{\mathbb {S}}_{jk}x_jx_k, \end{aligned}$$
(5.1)

for some symmetric matrix \({\mathbb {S}}.\) Define the projection:

$$\begin{aligned} \begin{aligned} \pi :\qquad \qquad \widehat{\mathcal {S}}\qquad \qquad&\longrightarrow \mathbb {H}^n \times \mathrm{Im}\,\mathbb {H}\simeq \mathbb {R}^{4n+3},\\(q_1,\ldots ,q_n, \phi (q')+\mathbf {t})&\longmapsto (q_1,\ldots ,q_n,\mathbf {t}), \end{aligned} \end{aligned}$$
(5.2)

where \(\mathbf {t}=t_1\mathbf i +t_2\mathbf j +t_3\mathbf k ,\)\(q_{l+1}=x_{4l+1}+\mathbf i x_{4l+2}+\mathbf j x_{4l+3}+\mathbf k x_{4l+4},\)\(l=0,\ldots ,n-1\) and \(t_\beta =x_{4n+1+\beta }\) for \(\beta =1,2,3.\) Let \(\psi :\mathbb {H}^{n}\times \mathrm{Im}\,\mathbb {H}\longrightarrow \mathcal {S}\subset \mathbb {H}^{n+1}\) be its inverse. The Cauchy–Fueter operator is

$$\begin{aligned} \overline{\partial }_{q_{l+1}}=\partial _{x_{4l+1}}+\mathbf i \partial _{x_{4l+2}}+\mathbf j \partial _{x_{4l+3}}+\mathbf k \partial _{x_{4l+4}}. \end{aligned}$$

Then, \(\overline{\partial }_{q_{l+1}}+\overline{\partial }_{q_{l+1}}\phi \cdot \overline{\partial }_{q_{n+1}}\) is a vector field tangential to the hypersurface \(\widehat{\mathcal {S}},\) since

$$\begin{aligned} \left( \overline{\partial }_{q_{l+1}}+\overline{\partial }_{q_{l+1}}\phi \cdot \overline{\partial }_{q_{n+1}}\right) \rho =0, \end{aligned}$$

\(l=0,1,\ldots ,n-1.\) This vector field is exactly the pushforward vector field \(\psi _*\big (\overline{\partial }_{q_{l+1}}+\overline{\partial }_{q_{l+1}}\phi \cdot \overline{\partial }_{\mathbf {t}}\big )\), where \(\overline{\partial }_{\mathbf {t}}=\mathbf i {\partial }_{t_{1}}+ \mathbf j {\partial }_{t_{2}}+\mathbf k {\partial }_{t_{3}}.\) Because

$$\begin{aligned} \psi _*\partial _{t_{\beta }}=\partial _{x_{4n+1+\beta }},\quad \psi _*\partial _{x_{4l+j}}=\partial _{x_{4l+j}}+\partial _{x_{4l+j}}\phi \cdot \partial _{x_{4n+1}}, \end{aligned}$$

for \(\beta =1,2,3,j=1,\ldots ,4,l=0,\ldots ,n-1 \), and

$$\begin{aligned} \begin{aligned} \psi _*\left( \overline{\partial }_{q_{l+1}}+\overline{\partial }_{q_{l+1}}\phi \cdot \overline{\partial }_{\mathbf {t}}\right)&=\sum _{j=1}^4\mathbf i _{j-1}\left( {\partial }_{x_{4l+j}}+{\partial }_{x_{4l+j}}\phi \cdot {\partial }_{x_{4n+1}}\right) \\ {}&\quad +\, \overline{\partial }_{q_{l+1}}\phi \left( \mathbf i {\partial }_{x_{4n+2}}+ \mathbf j {\partial }_{x_{4n+3}}+\mathbf k {\partial }_{x_{4n+4}}\right) = \overline{\partial }_{q_{l+1}}+\overline{\partial }_{q_{l+1}}\phi \cdot \overline{\partial }_{q_{n+1}} . \end{aligned} \end{aligned}$$

Denote

$$\begin{aligned} X_{4l+1}+\mathbf {i}X_{4l+2}+\mathbf {j}X_{4l+3}+\mathbf {k}X_{4l+4}:=\overline{\partial }_{q_{l+1}}+\overline{\partial }_{q_{l+1}}\phi \cdot \overline{\partial }_{\mathbf {t}}. \end{aligned}$$
(5.3)

Proposition 5.1

We have

$$\begin{aligned} X_{b}=\partial _{x_{b}}+2\sum _{\beta =1}^3\sum _{a=1}^{4n}\left( {\mathbb {S}}{\mathbb {I}}^\beta \right) _{ab}x_a\partial _{t_\beta }, \end{aligned}$$

where \({\mathbb {I}}^\beta \) is the \((4n)\times (4n)\) matrix \(\mathrm{diag}\left( I^\beta ,\ldots ,I^\beta \right) .\)

Proof

The proof is similar to that of Proposition 2.1 in [39]. Consider right multiplication by \(\mathbf i _\beta \). Note that

$$\begin{aligned} \begin{aligned} (x_1+x_2\mathbf i +x_3\mathbf j +x_4\mathbf k )\mathbf i =-x_2+x_1\mathbf i +x_4\mathbf j -x_3\mathbf k ,\\ (x_1+x_2\mathbf i +x_3\mathbf j +x_4\mathbf k )\mathbf j =-x_3-x_4\mathbf i +x_1\mathbf j +x_2\mathbf k ,\\ (x_1+x_2\mathbf i +x_3\mathbf j +x_4\mathbf k )\mathbf k =-x_4+x_3\mathbf i -x_2\mathbf j +x_1\mathbf k , \end{aligned} \end{aligned}$$

we can write

$$\begin{aligned} (x_1+x_2\mathbf i +x_3\mathbf j +x_4\mathbf k )\mathbf i _\beta&=-\,(I^\beta x)_1-(I^\beta x)_2\mathbf i -(I^\beta x)_3\mathbf j -(I^\beta x)_4\mathbf k \nonumber \\&=-\,\sum _{j=1}^4(I^\beta x)_j\mathbf i _{j-1}, \end{aligned}$$
(5.4)

where \(I^\beta \)’s are given by (2.41). \(B^\beta \) in (2.2) is the matrix associated with left multiplication by \(\mathbf {i}_\beta \) ([39, p. 1358]). Then, we have

$$\begin{aligned} \begin{aligned} \overline{\partial }_{q_{l+1}}\phi \cdot \partial _{\mathbf {t}}&=\left( \partial _{x_{4l+1}}\phi +\mathbf i \partial _{x_{4l+2}}\phi + \mathbf j \partial _{x_{4l+3}}\phi +\mathbf k \partial _{x_{4l+4}}\phi \right) \left( \mathbf i \partial _{t_{1}}+ \mathbf j \partial _{t_{2}}+\mathbf k \partial _{t_{3}}\right) \\ {}&= -\,\sum _{\beta =1}^3 \sum _{j,k=1}^{4}I_{jk}^\beta \partial _{x_{4l+k}}\phi \mathbf i _{j-1} \partial _{t_{\beta }}. \end{aligned} \end{aligned}$$

Substitute it into (5.3) to get

$$\begin{aligned} X_{4l+j}=\partial _{x_{4l+j}}+2\sum _{\beta =1}^3\sum _{k=1}^{4}\sum _{a=1}^{4n}I_{kj}^\beta {\mathbb {S}}_{a(4l+k)}x_a\partial _{t_\beta }=\partial _{x_{4l+j}}+2\sum _{\beta =1}^3\sum _{a=1}^{4n} \left( {\mathbb {S}}{\mathbb {I}}^\beta \right) _{a(4l+j)}x_a\partial _{t_\beta }, \end{aligned}$$

by the antisymmetry of \(I^\beta .\)\(\square \)

By Proposition 5.1, we get

$$\begin{aligned}{}[X_{a},X_{b}]=2\sum _{\beta =1}^{3}\left( \left( \mathbb {S}{\mathbb {I}}^\beta \right) _{ab}-\left( \mathbb {S}{\mathbb {I}}^\beta \right) _{ba}\right) \partial _{t_\beta }. \end{aligned}$$

So \(\mathrm{span}_\mathbb {C}\big \{X_1,\ldots ,X_{4n},\partial _{t_1}, \partial _{t_2},\partial _{t_3}\big \}\) is a nilpotent Lie algebra with center \(\mathrm{span}_\mathbb {C}\big \{\partial _{t_1}, \partial _{t_2},\partial _{t_3}\big \}.\) The corresponding nilpotent Lie group of step two is the group associated with the quadratic hypersurface \(\widehat{\mathcal {S}}\).

Now, if we choose the matrix \({\mathbb {S}}\) so that

$$\begin{aligned} \mathbb {S}{\mathbb {I}}^\beta +{\mathbb {I}}^\beta \mathbb {S}=2{\mathbb {B}}^\beta ,\quad \mathrm{where}\quad {\mathbb {B}}^\beta =\mathrm{diag}\left( B^\beta ,\ldots ,B^\beta \right) , \end{aligned}$$

then the Lie algebra spanned by \(X_1,\ldots ,X_{4n},\partial _{t_1},\partial _{t_1},\partial _{t_3}\) is isomorphic to the Lie algebra of the right quaternionic Heisenberg group \({\mathscr {H}}.\) It is sufficient to choose \(\mathbb {S}=\mathrm{diag}(S,\ldots ,S)\) such that \(SI^\beta +I^\beta S=2B^\beta ,\) where S is a symmetric \(4\times 4\) matrix. Namely,

$$\begin{aligned} C^\beta -\left( C^\beta \right) ^t=2B^\beta , \end{aligned}$$
(5.5)

for \(C^\beta =SI^\beta .\) Then,

$$\begin{aligned} S=\mathrm{diag}(-3,1,1,1 ), \end{aligned}$$

and

$$\begin{aligned} \begin{aligned} C^{1}&:=\left( \begin{array}{c@{\quad }c@{\quad }c@{\quad }c} 0 &{}\quad -\,3 &{}\quad 0 &{}\quad 0\\ -\,1&{}\quad 0&{}\quad 0&{}\quad 0\\ 0&{}\quad 0&{}\quad 0&{}\quad -\,1\\ 0 &{}\quad 0&{}\quad 1 &{}\quad 0 \end{array}\right) , C^{2}:=\left( \begin{array}{c@{\quad }c@{\quad }c@{\quad }c} 0 &{}\quad 0 &{}\quad -\,3 &{}\quad 0\\ 0&{}\quad 0&{}\quad 0&{}\quad 1\\ -\,1&{}\quad 0&{}\quad 0&{}\quad 0\\ 0 &{}\quad -\,1&{}\quad 0 &{}\quad 0 \end{array}\right) , \\ C^{3}&:=\left( \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }} 0 &{}\quad 0 &{}\quad 0 &{}\quad -\,3\\ 0&{}\quad 0&{}\quad -\,1&{}\quad 0\\ 0&{}\quad 1&{}\quad 0&{}\quad 0\\ -\,1 &{}\quad 0&{}\quad 0 &{}\quad 0 \end{array}\right) . \end{aligned} \end{aligned}$$

satisfy (5.5). Thus, the defining function (5.1) of \(\widehat{{\mathcal {S}}}\) in this case is (1.2) of \(\mathcal {S}\), and so the Lie group associate with \({\mathcal {S}}\) is the right quaternionic Heisenberg group.

5.2 The restriction of the k-Cauchy–Fueter operator

\(X_a\)’s for \(\mathcal {S}\) has the form

$$\begin{aligned} X_{4l+j}= \partial _{x_{4l+j}}+2\sum _{\beta =1}^3\sum _{k=1}^{4 } C_{kj}x_{4l+k}\partial _{t_\beta }. \end{aligned}$$

Since \(C^\beta \) is not antisymmetric, they are different from the standard left invariant vector fields (2.4) on \(\mathscr {H} \). It is standard that they can be transformed to the standard left invariant vector fields (2.4) on \(\mathscr {H} \) by a simple coordinate transformation \(\mathcal {F}:\mathscr {H}\rightarrow \mathbb {R}^{4n+3},(y,s)\mapsto (x,t)\) given by

$$\begin{aligned} x_{4l+j}=y_{4l+j},\quad t_\beta =s_\beta +\sum _{k,j=1}^4D^\beta _{kj}y_{4l+k}y_{4l+j}, \end{aligned}$$
(5.6)

(cf. [39, (1.8)]) with \( D^\beta :=C^\beta +\left( C^\beta \right) ^t \) symmetric. It is direct to see that

$$\begin{aligned} \mathcal {F}_*\partial _{s_\beta }=\partial _{t_\beta }\qquad \mathrm{and} \qquad \mathcal {F}_*Y_{4l+j}=X_{4l+j}, \end{aligned}$$

where \(Y_{4l+j}\) is given by (2.4). Then, we find the relationship between complex horizontal vector fields \(Z_A^{A'}\)’s on \(\mathscr {H} \) and \(\nabla _A^{A'}\)’s on \( {\mathbb {H}}^{n+1}\).

Proposition 5.2

Under the diffeomorphism \(\psi \circ \mathcal {F}:\mathscr {H}\rightarrow \mathcal {S}\), we have

$$\begin{aligned} \left( \psi \circ \mathcal {F}\right) _*Z_A^{A'} =\nabla _A^{A'}+\sum _{\alpha =0,1}C_A^{\alpha }\nabla _{(2n+\alpha )}^{A'},\quad \mathrm{for}\ \left( C_A^{\alpha }\right) :=\left( {\begin{matrix}\vdots \\ \tau \left( \overline{\partial }_{q_l}\phi \right) \\ \vdots \end{matrix}}\right) , \end{aligned}$$
(5.7)

for fixed \(A=0,1,\ldots ,2n-1,A'=0',1' \), where \(\tau \) is the embedding given by (2.13).

Proof

As \(\tau \) is a representation, we have

$$\begin{aligned} \begin{aligned} \psi _*&\left( \begin{array}{ll}-X_{4l+3}-\mathbf {i}X_{4l+4} &{}-X_{4l+1}-\mathbf {i}X_{4l+2} \\ \ \ X_{4l+1}-\mathbf {i}X_{4l+2} &{} -X_{4l+3}+\mathbf {i}X_{4l+4}\ \end{array}\right) =\psi _* \left( \begin{array}{rr}X_{4l+1}+\mathbf {i}X_{4l+2} &{} -X_{4l+3}-\mathbf {i}X_{4l+4} \\ X_{4l+3}- \mathbf {i}X_{4l+4}&{} X_{4l+1}-\mathbf {i}X_{4l+2} \end{array}\right) \varepsilon \\ {}&= \tau \left( \psi _*(X_{4l+1}+\mathbf {i}X_{4l+2}+\mathbf {j}X_{4l+3} +\mathbf {k}X_{4l+4})\right) \varepsilon \\ {}&= \tau \left( \overline{\partial }_{q_{l+1}}+\overline{\partial }_{q_{l+1}}\phi \cdot \overline{\partial }_{q_{n+1}}\right) \varepsilon =\tau \left( \overline{\partial }_{q_{l+1}}\right) \varepsilon +\tau \left( \overline{\partial }_{q_{l+1}} \phi \right) \tau \left( \overline{\partial }_{q_{n+1}}\right) \varepsilon \\ {}&=\left( \begin{array}{cc} \nabla _{(2l)}^{0'}&{}\nabla _{(2l)}^{1'}\\ \nabla _{(2l+1)}^{0'}&{}\nabla _{(2l+1)}^{1'} \end{array}\right) +\tau (\overline{\partial }_{q_l} \phi )\left( \begin{matrix}\nabla _{(2n)}^{0'}&{}\nabla _{(2n)}^{1'}\\ \nabla _{(2n+1)}^{0'}&{}\nabla _{(2n+1)}^{1'}\end{matrix}\right) , \end{aligned} \end{aligned}$$

where \(\varepsilon =\left( \begin{matrix}0&{}\quad -\,1\\ 1&{}\quad 0\end{matrix}\right) \) in (2.15). Then, (5.7) follows. \(\square \)

From this proposition, we can derive the relationship between operators in k-Cauchy–Fueter complex on \(\mathbb {H}^{n+1}\) and that in the tangential k-Cauchy–Fueter complex on \(\mathscr {H} \).

Proposition 5.3

Suppose that f is a k-regular function near \(q_0\in \mathcal {S}.\) Then, \(\left( \psi \circ \mathcal {F}\right) ^*f\) is k-CF on \(\mathscr {H}\) near the point \(\mathcal {F}^{-1}(\pi (q_0)) \).

Proof

As f is a k-regular function near \(q_0\in \mathcal {S}\subset \mathbb {H}^{n+1},\) we have \(\sum _{B'=0',1'}\nabla _{A}^{B'}f_{B' A_2'\ldots A_k'}=0\) for any fixed \(A =0,1,\ldots ,2n+1, \)\(A_2',\ldots , A_k' =0',1'.\) Then, we find that

$$\begin{aligned} \begin{aligned} \left. \left( \mathscr {D}_{0}(\psi \circ \mathcal {F})^*f\right) _{A A_2'\ldots A_k'} \right| _{\mathcal {F}^{-1}(\pi (q_0))}&=\sum _{B'=0',1'}\left. Z_{A}^{B'}((\psi \circ \mathcal {F})^*f)_{B' A_2'\ldots A_k'}\right| _{\mathcal {F}^{-1}(\pi (q_0))}\\ {}&= \sum _{B'=0',1'}\left( \psi \circ \mathcal {F}\right) _*Z_{A}^{B'}f_{B'A_2'\ldots A_k'}(q_0) \\&=\sum _{B'}\left( \nabla _{A}^{B'}+\sum _{\alpha =0,1}C_A^{\alpha } \nabla _{(2n+\alpha )}^{B'}\right) f_{B' A_2'\ldots A_k'}(q_0)=0, \end{aligned} \end{aligned}$$

for any fixed \(A =0,1,\ldots ,2n-1, \)\(A_2',\ldots , A_k' =0',1' \), by Proposition 5.2. The proposition is proved. \(\square \)