Update! Please see my new blog post about setting up the Magento cron system to get the most out of Magento and bypass major headaches in understanding how Magento works with cron schedules.
Magento cron tasks are very useful for scheduling automatic updates to your Magento install such as running automatic Google sitemap generation and currency rate updates, and since Magento 1.9.2 transactional emails are also delivered via a cron task. However, the Magento wiki documentation for cron scheduling is confusing and there are numerous forum topics relating to Magento cron errors. The main problem that I hope to have found a resolution to involves the Magento cron_schedule table.
From my experiments I have found that cron jobs ran using the default settings and a cron job interval of 15 minutes, as stated in the wiki documentation, become a real resource hog and actually overload the cron system so that new scheduled tasks are spawned before predecessors tasks have a chance to complete!
So my solution is straightforward and simple: Don’t schedule too far into the future (if at all) and make sure you clear failed and successful cron schedules at regular intervals.
So here is my set up:
Login to your Magento admin interface and go to: system > configuration > system. Then using the ‘Cron (Scheduled Tasks)’ tab enter the following information:
Generate schedules every: 15
Schedule ahead for: 20
Missed if not run within: 15
History cleanup every: 10
Success history lifetime: 60
Failure history lifetime: 600
You can then set cron jobs to run at virtually any interval. I have mine set to 15 minutes for most of my clients due to their use of custom modules related to scheduled tasks.
Please note: This is an experimental technique that has been deployed to small number of clients so please report any problems you have and I will update this post accordingly.
Further, the Magento support team have stated that cron jobs should be run no more than once a day. I hope to bring a little clarity to this situation because the only viable reason why cron tasks should be run at this interval is due to a lack of cron configuration for catalogue indexing and newsletter sending. Both of these modules are configured by default to execute each time the cron job is run, hence cron_schedule overload!