The next time your horse is drifting sideways, try this on the side the horse is drifting to.
Either one (both) of you may be unbalanced and this will help while you work on correcting the imbalance in your horse.
Imagine you are sitting (seatbone), and standing (stirrup), on a thumbtack.
To not get poked, you need to slightly lift your weight off the thumbtack. It is very subtle. No one should see you make a change. Just don't let that thumbtack poke you.
See if that helps your horse to not drift as much.
Bonus tip: If you want the horse to move sideways (leg-yield). Imagine the thumbtack is on the opposite side of where you want to go. The increase of weight in the direction of travel will help the horse move that way to get back underneath you.
In both cases, if you don't notice a difference don't make the movement bigger. It means there are other things you need to identify and correct. That is what we're going to work on in this course.