Are you using a fresh towel every day, separate from the communal bathroom hand towel? When you use the same towel to dry your face as is used to dry hands throughout the day you risk transferring those bacteria onto your facial skin, potentially causing infections or irritation.
Okay, you have a specific towel just for your face, but how often is it being washed? Towel fibers hold onto water, which then sits and creates mold and bacteria. Not to mention, any dirt and bacteria that transferred from your cleansing routine are now on the towel, ready to transfer back to your skin the next time you use it.
What about the laundry detergent you are using to clean your towels? It is not uncommon for residue from laundry detergents to remain on towels even after washing, causing irritation when coming into contact with the skin. Detergents can also be drying - stripping the skin of its natural oils.
The best practice is a fresh towel every day for your morning and then night routine separate from the communal bathroom towel. Use a fragrance-free, hypoallergenic detergent to wash your towels.