Listening Early Goes A Long Way