To be fair, this happens to me also in Debian Gnome, but very very rarely. On Windows, it keeps me from ejecting the drive until after I try it 4-5 times. Every time. In any case, lsof can help identify the culprit.
lsof | grep DRIVENAME
should give which program is keeping it.
https://www.imdb.com/title/tt0413623/