Average partial effects (APEs) are generally not point-identified in binary response panel models with unrestricted unobserved heterogeneity. We show their point-identification under an index sufficiency assumption on the unobserved heterogeneity, even when the error distribution is unspecified. This assumption does not impose parametric restrictions on the unobserved heterogeneity. We then construct a three-step semiparametric estimator for the APE. In the first step, we estimate the common parameters using either a conditional logit or smoothed maximum score estimator. In the second step, we estimate the conditional expectation of the outcomes given the indices and a generated regressor that depends on first-step estimates. In the third step, we average derivatives of this conditional expectation to obtain a partial mean that estimates the APE. We show that this proposed three-step APE estimator is consistent and asymptotically normal. We evaluate its finite-sample properties in Monte Carlo simulations. We then illustrate our estimator in a study of determinants of married women’s labor supply.