Washington was referring to his military family or aides-de-camp, the same way John Adams described the aide Alexander Hamilton as “one of General Washington’s Family.” So when Washington said “family,” he meant “chummy minion.” The orphaned Lafayette heard “son.
