Should I worry?

As long as you are taking your pill at a regular time you should be ok but check with your doctor as well for additional information to see what they say.