Absolute values of the quasiparticle energies of small silicon clusters (Sin, n=4,5,6) are determined quite accurately by means of the state-of-the-art ab initio calculations in the GW approximation, under the all-electron mixed-basis representation. We find that the electron affinity, which is very sensitive to the ionic valence and the cluster geometry, depends strongly on the photoemission process and that the time-reversal symmetry is completely broken between the photemission and its inverse process. This is due to large structural changes in Si5- and Si6- as compared to the neutral clusters. When appropriate structural energy changes are taken into account, the time-reversal symmetry is satisfied and all results agree excellently with the preexisting experimental data. {\textcopyright} 2003 The American Physical Society.