True - it would be interesting to use, say, the coding layer of an autoencoder trained on one of the many face datasets and then training only on the last few layers to fit their data.