Change the locks! Then if he sneaks back in he is trespassing and you can get him escorted off the premises, and if needed apply for a restraining order. It sounds like he has been taking advantage of your too-nice nature for awhile now!
Or, you could move & not tell him where you went........

You don’t say whether it is your place, but I am assuming that it is, so be strong and tell him straight you need to have the place to yourself. The brother is one thing, as he is family you will still want to have some sort of a relationship with him, but the girlfriend is another matter. Once he goes change the locks and tell him he can only come on visits when invited, but no girlfriend. Be firm, stop the cycle, reclaim your own territory. My adult son came back and lived with me for a while a few years ago.....it was hell! I love him to bits, but his habits around cooking, especially late at night ( garlic wafting upstairs to my bedroom at midnight ), and the hours he was coming in and going out at nearly drove me insane. Now he lives an hour away and we visit each other on a regular basis, all is harmony once again!

My sister lived with me after she orchestrated the demise of her own life (lost jobs, quit jobs, foreclosed house). She tried to create havoc in my house. Rules were clear & when the time passed that she said she'd be out, I gave her 30 days to get out. She called me pretty heinous names, left an ungodly mess but left all the same.

Check laws in your state. You may have to go through a formal eviction process.