For families who don’t need to borrow, higher rates might not affect daily life. But for those who do, higher rates are really starting to hurt.