Agreed: your concern ought to be for “them”. But your first concern must be yourself, your own interior life. Otherwise, you will not be able to serve them.