We shouldn't even really say that we 'meditate' until we can actually get to the state of Dhyana. Technically, most of us mere mortals are just trying to put ourselves into a state that we're more likely to be able to slip into a meditation.
It's natural to have passing thoughts and the important thing is to let them pass, and go back to what your focus was on (e.g. breath, visualisation, a candle flame).
To put it into context, Patanjali's Yoga Sutra explores the 8 limbs of Ashtanga. This is a hierarchical guide to how you can live a good yogic life. Moving your body (Asana) and purposefully directing your breath (Pranayama) are merely limb 3 and 4. Dharana is limb 7 out of the 8, so we shouldn’t be surprised when it’s not instantly obtained just because we want it.
Rather than sitting there getting stressed because you can't clear your mind and jump straight into meditation, start with some Asana and Pranayama - you may only need 5 - 10 minutes to prepare yourself. You can then have the goal to relax and try to direct your mind onto a single point of focus and when your mind drifts just take it back, then when your mind drifts again take it back, and when it drifts again take it back, and back, and back...
Gradually, over many many practices and sometimes years, decades and maybe never, you can fall into meditation where you are one with whatever you are aiming to connect with and then who knows, maybe you can reach eternal bliss in Samadhi.
If meditation continues to allude you, then keep your yoga practice to Asana and Pranayama until you are more prepared to move towards Dhyana, via the steps of Pratyahara (withdrawal from external stimulus) and Dharana (withdrawal from internal distractions).
Ultimately, Yoga is supposed to be about the journey, so enjoy where you are and the benefits you're getting from your yoga practice.