2 1. INTRODUCTION

to C is degenerate, and we do not have a natural pseudo-Riemannian structure on

C. Consequently, there is no natural single operator on C such as the Laplace–

Beltrami operator. However, it turns out that there are commuting, self-adjoint,

second order differential operators P1,...,Pn that we call fundamental differential

operators on C satisfying the algebraic relation P1 2 +···+Pn1 2 −Pn1+1 2 −· · ·−Pn 2 = 0.

Then, what we want is to understand how an arbitrary function on C (of appropriate

class) is expanded into joint eigenfunctions of P1,...,Pn.

We will find explicit joint eigendistributions for P1,...,Pn, and construct a

(well-defined) transform, to be denoted by FC, by means of these eigenfunction.

The transform FC intertwines the multiplication by coordinate functions with the

differential operators Pj. Moreover, we prove that we can normalize FC such that it

is involutive, i.e. FC

2

= id and unitary. Thus, we establish its inversion formula and

the Plancherel type theorem. It is noteworthy that the kernel function K(x, x ) of

FC involves singular distributions (e.g. normal derivatives of Dirac’s delta function

with respect to a hypersurface) but yet that the operator FC is unitary in the

general case where n1,n2 1 and n1 + n2 4. In the case n1 = 1, n2 = 1 or

(n1,n2) = (2, 2), FC reduces to the Hankel transform composed by a (singular)

Radon transform.

The second motivation comes from representation theory of real reductive

groups, in particular, from minimal representations.

Minimal representations are infinite dimensional unitary representations that

are the ‘closest’ to the trivial one-dimensional representation. The Weil represen-

tation of the metaplectic group Mp(n, R), which plays a prominent role in the

construction of theta series, is a classic example. Most minimal representations are

isolated among the set of irreducible unitary representations, and cannot be built

up from the existing induction techniques of representation theory.

A multitude of different models of minimal representations have been investi-

gated recently by many people (see Sections 1.4 and 1.5). Each model known so

far has its own advantages indeed but also has some disadvantages. For instance,

the inner product of the Hilbert space is not explicit in some models, whereas the

whole group action is not clear in some other models.

A challenge to surmount that ‘disadvantage’ may turn up as a natural problem

in other areas of mathematics. In order to give its flavor, let us consider two

geometric models of minimal representations of the indefinite orthogonal group

G = O(n1 + 1,n2 + 1): One is in the solution space to the Yamabe equation

(conformal model), and the other is in

L2(C)

(Schr¨ odinger model).

In the conformal model, the whole group action is very clear, whereas the inner

product is not. The problem of finding the explicit inner product was solved in

the previous paper [48] as the theory of conserved quantities for ultra-hyperbolic

equations, such as the energy for the wave equation.

In the Schr¨ odinger model

L2(C),

the unitary structure is clear, whereas the

whole group action is not. The understanding of the whole group action was a

missing piece of [48]. This problem is reduced to finding the generalization of the

Fourier–Hankel transform on the isotropic cone C, namely, the above mentioned

operator

FC. By finding an explicit formula of FC, we shall settle this problem.

The role of FC in our minimal representation is in parallel to that of the Euclidean

Fourier transform FRn in the Weil representation, summarized as below: