Surveys, screeners, and patient assessments are often shortened to decrease response burden and cost of administration. While there are many methods for shortening screeners, they often aim to reproduce the original score while inducing the smallest amount of measurement error. One consideration that is often overlooked is that these instruments are often used less for precise measurement of some latent construct, but instead for accurate prediction of an even more costly or time consuming diagnosis or expert judgment. As such, we present an alternative method that addresses these concerns through the use of a Markov Chain Monte Carlo algorithm with simulated annealing (MCMC-SA). To maintain ease of use of the eventual form, we use MCMC-SA to explore the combinatorial search space of short forms using a loss function that optimizes prediction based upon an unweighted sum score. This method is orders of magnitude more efficient than brute force search and has the advantage of optimizing for prediction of the desired outcome directly, instead of optimizing on the measurement of an intermediate outcome that may not be well aligned with the decision criteria of interest. When applied to a screener for alcohol use disorder, we demonstrate that under the unweighted sum score scoring constraint, shortened forms can more accurately predict expert judgments than full forms.