WP All Import cron from CLI (update)

My previous post about executing the WP All Import cron jobs with PHP cli didn’t work too well. Apparently the processing part still stops executing after a while. I ended up with import jobs stating “last activity 5 hours ago”.

Still I don’t want to run this through the webserver. So I wrote a little wrapper shell script to execute my imports in cron. It checks if the line “is not triggered” was partially returned. The wp cron script would return “<p>Import #26 is not triggered. Request skipped.</p>” if you execute the processing action without triggering it first. In other words: the import is really complete now.

I put the shell script outside my document root, and it uses the “logs” directory to store the log files. So make sure this directory is writable and points to the right path.

Now just put this in your /etc/cron.d/yoursite:

 

15 thoughts on “WP All Import cron from CLI (update)

  1. hi glad i found you, as after trying a cron on a large file i am getting the same problem of “last activity 5 hours ago”.

    I wonder if you could help me a little, im a total noob?

    I see where i must edit the xxYourKeyxx, but not sure about where to define the JOB (7?) number AND also hwo to check the path to the log? Currently logs for the wp al import when i trigger the cron are written to the

    /public_html/wp-content/uploads/wpallimport/logs

    —– my host told me

    “/etc/cron.d/yoursite is the equivalent of the advanced mode editor accessible via the control pan”

    so i guess i can add this via

    /usr/bin/wget -O /dev/null -o /dev/null http://www.mydomain.com/wp-cron.php?import_key=MYKEY&import_id=7&action=processing

    /usr/bin/wget -O /dev/null -o /dev/null http://www.wmydomain.com/wp-cron.php?import_key=MYKEY&import_id=7&action=trigger

    help much appreciated, regards Lee

      • Yo roland can you help me out I need to use your cron again, but its stopped working if i give you shell access can you help me out, I trid the host, but is not helping. and access to backend wp can you check my setting for me. >?? can you email me an address to send the deets to if you have the time to take a look Cheeersss Lee :)

  2. Hi,

    I ran the import cron to run every 2 mins and the trigger cron to run every 1 hour.

    Now on the server root, the plugin seems to be creating countless files with the name wp-cron.php_import_key=[value]&import_id=12&action=processing.154….and so on….

    Each file has the text:
    Import #[number] is not triggered. Request skipped.
    and is 54-53 bytes in size. Any idea what could be causing this?

    Thanks much in advance.

    • Seems to me that this is the output of the script. I’m not sure how your cron is configured, and if you are using some kind of control panel. You could try adding to your crontab rule: >/dev/null 2?&1. So the complete line would be:

      0 6 * * * yourwebsite /home/yourwebsite/wpai_cron.sh -j 26 >/dev/null 2>&1

  3. Hey,

    Does your script still work?

    I put it the script in the directory: /var/www/domain.com/
    WordPress is in: /var/www/domain.com/htdocs/
    Logs directory: /var/www/domain.com/logs/ (permissions are set right)

    Also I did chmod +x on the script.
    When I try to execute it though, it outputs nothing. No output at all, and it never ends. I modified it in one line only and changed public_html -> htdocs

    Any ideas?

  4. Hi,

    I have a lot of problems with my wp all import crons. After a while a cron shows the message “last activity….” and the whole site becomes really slow.

    Is your script still valid? I don’t know coding, so I’m just trying to make the small changes like import key. :)

    I created the file wpai_cron.sh with your code above and inserted my import key.

    Then I have a cron like this
    0 6 * * * /home/fanatiqu/wpai_cron.sh -j 46

    However, it doesn’t work.

    Do I have to create a logs directory and where do I put it? Anything else that I miss?

    Your feedback on this would be highly appreciated since it causes a lot of problems.

    Have a great day!

    Best regards,
    Daniel

    • Hi Daniel,

      I don’t support the site anymore that was running this code. I am not sure if the script still works, and I am unable to test this. The shell script that I posted writes the output of the cron to a log file. So yes, you would need to create that directory or make sure you use one that can be written by the user that runs the cronjob. At line 26 and 27 you could change the log location to whatever location you would like.

      The output of the cron could give you hints why it’s not working.

      Good luck!

    • Hi Daniel, did you ever find a fix for this? WP ALL Import cron jobs completely crashed my site and too am needing a solution.

      Roland have you ever considered creating a plugin for this? I wish WP ALL Import would. My guess is that anyone who uses cron jobs for stock updates is going to run into this problem. Any insight is much appreciated.

      • Performance was really poor indeed. I remember I ended up writing my own script specifically for updating stock. This was not configurable at all but it did the job really fast.

        If you have a large product feed, then wp all import isn’t the best way to keep your stock updated. I think you’re better off if you develop a script on your own.

  5. Great Script! The only Problem is the speed. And i know it’s not because of your script. But i don’t know why.

    I have a csv with 8500 products.

    If i set the iterage to 20 and do the import via WP All Import Backend, it tooks only 1-2 hours.

    If i set the iterage to nothing (empty) and set to high speed and do the import via cli, it tooks more than 24 hours.

    dammed, where is the bottleneck?

    • Try setting iterage as high as possible. If it’s too high, you will run out of memory. So I guess you could try 50, 100, etc. I think I’d stop at 100 if the time is acceptable then.

      I haven’t used WP All Import since that post anymore, but indeed, I found the plugin pretty slow as well. Especially if you just want to update the stock information. Eventually I wrote my own script that would just update the stock. If you create multiple update statements that execute at once, this can be really fast. I remember the script only took a few seconds, with thousands of products.

      I think the bottleneck is because WP All Import is pretty complex because you can configure so many options. It’s a really cool plugin if you cannot code yourself, although I imagine it can be hard to get it working then. But because of this, I don’t think it works as efficient as possible. If you write custom code for your specific feed, you can be a lot faster.

  6. I am currently working on a site that requires daily updates to an xml file. This cron script looks ideal to what I want to do; however, I am confused (Newbie) to what this line below means. I am guessing the LOGFILE & CURLOG is the start dir from where ever the wpai_cron.sh is placed?

    # Set magic variables for current FILE & DIR
    __FILE__=”$(test -L “$0” && readlink “$0” || echo “$0″)”
    __DIR__=”$(cd “$(dirname “${__FILE__}”)”; echo $(pwd);)”
    LOGFILE=”${__DIR__}/logs/wpai_cron_${jobId}.log”
    CURLOG=”${__DIR__}/logs/wpai_cron_${jobId}_current.log”
    DONE=0

    Can you also explain what does ‘website’ and ’26’ represent?
    0 6 * * * yourwebsite /home/yourwebsite/wpai_cron.sh -j 26

    I am assuming ’26’ is the jobID? What do I input as the ‘yourwebsite’?

    Will this work?
    0 6 * * * /home/www/wpai_cron.sh -j 26

    • The LOGFILE and CURLOG point to files where the output of the cron is being written to. So this location needs to be writable to the crontab user. These logfiles are used to detect if the processing was complete. If you read the script, you see that it searches for the string is not triggered. That’s why it needs a current log, because in the large log this string will always be found.

      The cron definition was just an example. It depends on how your server is setup, but it needs to run under the same user as your webserver. The 26 is the job ID indeed. I’m not sure if your crontab definition works; I think it depends on where you define it. But I think there are enough articles on the internet that explain how define a crontab for your linux distro.

  7. Hi,
    I’ve got this up and running and it’s importing fine but there is a problem where the import never actually ends and untriggers. Any ideas of why this could be and any method of fixing it?

    I’ve tried setting up the cron to hit the sh file every 2 mins / 30 mins / hour / 24 hours, none of these options actually end it. So it ends up in a constant cycle of importing.

    I’m wondering if it’s due to the import rules I have set that tell it to not import if there is no image to the product being imported this results in the log files sticking at around 2500 of 4000 products imported as an example. I’m not sure what’s going on with it, any ideas?

    This is in the logs with the cron set to 30mins:

    {“status”:200,”message”:”Records Processed 2376. Records imported 2375 of 4214.”}Sat Aug 11 10:30:02 BST 2018: Start import for jobID 6
    Sat Aug 11 11:00:03 BST 2018: Start import for jobID 6
    Sat Aug 11 11:30:03 BST 2018: Start import for jobID 6

Leave a Reply

Your email address will not be published. Required fields are marked *