Contemporary Mathematics Volume 290, 2001 Eigenfunctions of the transfer operators and the period functions for modular groups Cheng-Hung Chang and Dieter H. Mayer ABSTRACT. We extend the transfer operator approach to the period functions of Lewis and Zagier for the group PSL(2, Z) [LZ97] to general subgroups of PSL(2, Z) with finite index. Thereby we derive functional equations for the eigenfunctions of the transfer operators with eigenvalue . = 1 which generalize the one derived by J. Lewis for PSL(2,Z). For special congruence subgroups we find that Lehner and Atkins theory of old and new forms [AL70] is realized also on the level of the eigenfunctions of the transfer operators for these groups. It turns out that the old eigenfunctions can be related through Lewis' transform for PSL(2, Z) to the old forms for these subgroups, whereas a similar relation for new eigenfunctions and the corresponding new forms is not yet known. 1. Introduction In the Eichler-Manin-Shimura theory of periods [Eic57] to every holomorphic modular cusp form one can associate a period polynomial with certain cocycle properties under the action of the group elements. This theory was extended to general holomorphic forms for PSL(2, 7!..) by Zagier in [Zag91]. Regularizing the corresponding integrals of Eichler in the case of the holomorphic Eisenstein series for PSL(2, 7!..) leads to certain rational functions. Recently, J. Lewis found a further extension of this theory in which to every MaaB wave form for PSL(2, 7!..) there is related a certain holomorphic function in the cut z-plane which fulfills a functional equation depending on a complex parameter {3. This equation combines the two cocycle relations for the polynomials of the two generators of PSL(2, 7!..) [Lew97], [LZ97], [LZ]. The polynomial solutions of this equation are just the period poly- nomials and its rational solutions are the rational functions of Zagier. Zagier found another discrete series of solutions of the Lewis-Zagier functional equation which are transforms of the nonholomorphic Eisenstein series of PSL(2, 7!..) for parameter values /3 corresponding to the resonances of the Laplacian on the modular surface [CM99] given by the nontrivial zeros of Riemann's zeta function (R(2/3). 1991 Mathematics Subject Classification. (MSC 2000) Primary 37C30, 81Q50, 11F67 Sec- ondary 11F66, 37D40, 11M36. This work is supported by DFG Schwerpunktprogramm 'Ergodentheorie, Analysis und ef- fiziente Simulation dynamischer Systeme'. © 2001 American Mathematical Society 1 http://dx.doi.org/10.1090/conm/290/04571

Purchased from American Mathematical Society for the exclusive use of nofirst nolast (email unknown) Copyright 2001 American Mathematical Society. Duplication prohibited. Please report unauthorized use to cust-serv@ams.org. Thank You! Your purchase supports the AMS' mission, programs, and services for the mathematical community.