I use it on cereal but I couldn't imagine using it as a substitute in tea or coffee.
Coconut milk is really nice in coffee (I don't drink tea, so can't comment on it).
It gets used no problem in coffee shops to make lattes and cappuccino, or you can add it cold to black coffee.
Almond milk one could make by using a juicer separator, expensive bit of kit though.
Oat milk is easily made by straining oats, water, vegetable oil that have been blended together.
Apparently, the oil is to make it creamier, can be substituted with cashew nuts.
I have tried to make oat milk, was good but became slimy on the second day.
Apparently, there are various tips for not getting a slimy result, but I have not yet investigated
I like all the milk alternatives, but I have noticed that Aldi's almond milk doesn't taste as good as Tesco's, while their own brand oat milk tastes better than a popular branded one.
Ime, all plant based milks work well as milk substitutes in baking.