After spending some time experimenting with the Magento cron system I now have some good results and some helpful hints from a well placed source. During my tests I setup a cron command to launch the Magento cron.php file at intervals of 5, 10, 15, 30 and 60 minutes.
Here is what I have found out:
- Magento spends around 1 minute to index 1000 products.
- Look at your cron_schedule table in the database and make sure that scheduled tasks are NOT overlapping - each task must have a status 'success' before a new task is launched with status 'pending'.
This is simple guidance that will allow you to make an informed decision concerning the Magento scheduled task settings at: system > configuration > system > ‘Cron (Scheduled Tasks)’ tab.
Here are the settings I am currently using for some of my clients stores:
Generate schedules every: 15
Schedule ahead for: 20
Missed if not run within: 15
History cleanup every: 10
Success history lifetime: 60
Failure history lifetime: 600
My cron job is set to run every fifteen minutes.
The above settings have been configured to allow Magento to generate and clean schedules within a 2-hour time frame.
Please note: I have my reservations about the Magento cron system in its default state because a few core modules have cron configurations that will be initiated on each and every cron jon run which leads to a bloated cron_schedule table and overlapping cron schedules, 'success/pending' conflicts. Mainly the conflicts arise from the Magento job codes labelled 'catalogindex_run_queued' and 'newsletter_send_all' in the cron_schedule table. These schedules originate from their associated modules, using dreamweaver or textpad do a search for <schedule><cron_expr> and you will see the problem with the CatalogIndex and Newsletter module's config.xml files. The 2 config files have schedules that are set to initiate on each cron run, therefore you have the potential problem that schedules will overlap causing the Magento cron system to fail.
The problem could be resolved by making a custom module for these cron settings. However, it would be much more flexible to have the schedule settings available within the admin interface for the CatalogIndex and Newsletter modules. So, until this situation is resolved use either a custom module or scale back your usage of cron jobs to allow catalog indexing to complete with a status of 'success' in the cron_schedule table.