CHAPTER 1

Introduction

A representation π of a group G defines a representation of a subgroup G by

restriction. In general irreducibility is not preserved by the restriction. If G is

compact then the restriction π|G is isomorphic to a direct sum of irreducible rep-

resentations π of G with multiplicities m(π, π ). These multiplicities are studied

by using combinatorial techniques. If G is not compact and the representation π

is infinite-dimensional, then generically the restriction π|G is not a direct sum of

irreducible representations [16] and we have to consider another notion of multi-

plicity.

For a continuous representation π of G on a complete, locally convex topological

vector space Hπ, the space Hπ

∞

of

C∞-vectors

of Hπ is naturally endowed with a

Fr´ echet topology, and (π, Hπ) gives rise to a continuous representation

π∞

of G on

Hπ

∞.

If Hπ is a Banach space, then the Fr´ echet representation

(π∞,

Hπ

∞)

depends

only on the underlying (g,K)-module (Hπ)K. Given another continuous represen-

tation π of the subgroup G , we consider the space of continuous G -intertwining

operators (symmetry breaking operators)

HomG

(π∞|G

, (π

)∞).

The dimension m(π, π ) of this space yields important information of the restriction

of π to G and is called the multiplicity of π occurring in the restriction π|G . Notice

that the multiplicity m(π, π ) makes sense for non-unitary representations π and

π , too. In general, m(π, π ) may be infinite. For detailed analysis on symmetry

breaking operators, we are interested in the case where m(π, π ) is finite. The

criterion in [25] asserts that the multiplicity m(π, π ) is finite for all irreducible

representations π of G and all irreducible representations π of G if and only if

the minimal parabolic subgroup P of G has an open orbit on the real flag variety

G/P , and that the multiplicity is uniformly bounded with respect to π and π if

and only if a Borel subgroup of GC has an open orbit on the complex flag variety

of GC.

The classification of reductive symmetric pairs (g, g ) satisfying the former con-

dition was recently accomplished in [20]. On the other hand, the latter condition

depends only on the complexified pairs (gC, gC), for which the classification is much

simpler and was already known in 1970s by Kr¨ amer [28]. In particular, the multi-

plicity m(π, π

) is uniformly bounded if the Lie algebras (g, g ) of (G, G ) are real

forms of (sl(N + 1, C), gl(N, C)) or (o(N + 1, C), o(N, C)).

In this article we confine ourselves to the case

(1.1) (G, G ) = (O(n + 1,1),O(n,1)),

1