Would I be unfair to my brother?

Maybe you can go to live with your aunt for a while while your brother is financially stable enough to take care of all of you. It might be hard at first, but just think that all of you will be together at the end :)