2 1. INTRODUCTION
to C is degenerate, and we do not have a natural pseudo-Riemannian structure on
C. Consequently, there is no natural single operator on C such as the Laplace–
Beltrami operator. However, it turns out that there are commuting, self-adjoint,
second order differential operators P1,...,Pn that we call fundamental differential
operators on C satisfying the algebraic relation P1 2 +···+Pn1 2 −Pn1+1 2 −· · ·−Pn 2 = 0.
Then, what we want is to understand how an arbitrary function on C (of appropriate
class) is expanded into joint eigenfunctions of P1,...,Pn.
We will find explicit joint eigendistributions for P1,...,Pn, and construct a
(well-defined) transform, to be denoted by FC, by means of these eigenfunction.
The transform FC intertwines the multiplication by coordinate functions with the
differential operators Pj. Moreover, we prove that we can normalize FC such that it
is involutive, i.e. FC
= id and unitary. Thus, we establish its inversion formula and
the Plancherel type theorem. It is noteworthy that the kernel function K(x, x ) of
FC involves singular distributions (e.g. normal derivatives of Dirac’s delta function
with respect to a hypersurface) but yet that the operator FC is unitary in the
general case where n1,n2 1 and n1 + n2 4. In the case n1 = 1, n2 = 1 or
(n1,n2) = (2, 2), FC reduces to the Hankel transform composed by a (singular)
The second motivation comes from representation theory of real reductive
groups, in particular, from minimal representations.
Minimal representations are infinite dimensional unitary representations that
are the ‘closest’ to the trivial one-dimensional representation. The Weil represen-
tation of the metaplectic group Mp(n, R), which plays a prominent role in the
construction of theta series, is a classic example. Most minimal representations are
isolated among the set of irreducible unitary representations, and cannot be built
up from the existing induction techniques of representation theory.
A multitude of different models of minimal representations have been investi-
gated recently by many people (see Sections 1.4 and 1.5). Each model known so
far has its own advantages indeed but also has some disadvantages. For instance,
the inner product of the Hilbert space is not explicit in some models, whereas the
whole group action is not clear in some other models.
A challenge to surmount that ‘disadvantage’ may turn up as a natural problem
in other areas of mathematics. In order to give its flavor, let us consider two
geometric models of minimal representations of the indefinite orthogonal group
G = O(n1 + 1,n2 + 1): One is in the solution space to the Yamabe equation
(conformal model), and the other is in
(Schr¨ odinger model).
In the conformal model, the whole group action is very clear, whereas the inner
product is not. The problem of finding the explicit inner product was solved in
the previous paper  as the theory of conserved quantities for ultra-hyperbolic
equations, such as the energy for the wave equation.
In the Schr¨ odinger model
the unitary structure is clear, whereas the
whole group action is not. The understanding of the whole group action was a
missing piece of . This problem is reduced to finding the generalization of the
Fourier–Hankel transform on the isotropic cone C, namely, the above mentioned
FC. By finding an explicit formula of FC, we shall settle this problem.
The role of FC in our minimal representation is in parallel to that of the Euclidean
Fourier transform FRn in the Weil representation, summarized as below: