WP All Import cron from CLI (update)

Written by

My previous post about executing the WP All Import cron jobs with PHP cli didn’t work too well. Apparently the processing part still stops executing after a while. I ended up with import jobs stating “last activity 5 hours ago”.

Still I don’t want to run this through the webserver. So I wrote a little wrapper shell script to execute my imports in cron. It checks if the line “is not triggered” was partially returned. The wp cron script would return “<p>Import #26 is not triggered. Request skipped.</p>” if you execute the processing action without triggering it first. In other words: the import is really complete now.

I put the shell script outside my document root, and it uses the “logs” directory to store the log files. So make sure this directory is writable and points to the right path.

#!/bin/bash
 
while getopts “:j:” opt; do
  case $opt in
    j)
      jobId=$OPTARG
      ;;
    \?)
      echo “Invalid option: -$OPTARG” >&2
      ;;
   :)
      echo “Option -$OPTARG requires an argument.” >&2
      exit 1
      ;;
  esac
done

if [ “$jobId” = “” ]; then
  echo No job id
  exit 1
fi

# Set magic variables for current FILE & DIR
__FILE__=”$(test -L “$0” && readlink “$0” || echo “$0″)”
__DIR__=”$(cd “$(dirname “${__FILE__}”)”; echo $(pwd);)”
LOGFILE=”${__DIR__}/logs/wpai_cron_${jobId}.log”
CURLOG=”${__DIR__}/logs/wpai_cron_${jobId}_current.log”
DONE=0

function log {
  echo “$(date): $*” >>$LOGFILE
}


log “Start import for jobID $jobId”

cd $__DIR__/public_html
php -e -r ‘parse_str(“import_key=xxYourKeyxx&import_id=’$jobId’&action=trigger”, $_GET); include “wp-cron.php”;’ >>$LOGFILE 2>&1
sleep 1

while [ $DONE -eq 0 ]
do
  php -e -r ‘parse_str(“import_key=xxYourKeyxx&import_id=’$jobId’&action=processing”, $_GET); include “wp-cron.php”;’ >$CURLOG 2>&1
  cat $CURLOG >>$LOGFILE
  DONE=$(grep ‘is not triggered’ $CURLOG | wc -l)
  sleep 1
done
rm $CURLOG

log “End of import for jobId $jobId”
log “”
log “”

#!/bin/bash

while getopts “:j:” opt; do

case $opt in

jobId=$OPTARG

;;

\?)

echo “Invalid option: -$OPTARG” >&2

;;

echo “Option -$OPTARG requires an argument.” >&2

exit 1

;;

esac

done

if [ “$jobId” = “” ]; then

echo No job id

exit 1

# Set magic variables for current FILE & DIR

__FILE__=“$(test -L “$0” && readlink “$0” || echo “$0“)”

__DIR__=“$(cd “$(dirname “${__FILE__}”)“; echo $(pwd);)”

LOGFILE=“${__DIR__}/logs/wpai_cron_${jobId}.log”

CURLOG=“${__DIR__}/logs/wpai_cron_${jobId}_current.log”

DONE=0

function log {

echo “$(date): $*” >>$LOGFILE

}

log “Start import for jobID $jobId”

cd $__DIR__/public_html

php –e –r ‘parse_str(“import_key=xxYourKeyxx&import_id=’$jobId‘&action=trigger”, $_GET); include “wp-cron.php”;’ >>$LOGFILE 2>&1

sleep 1

while [ $DONE –eq 0 ]

php –e –r ‘parse_str(“import_key=xxYourKeyxx&import_id=’$jobId‘&action=processing”, $_GET); include “wp-cron.php”;’ >$CURLOG 2>&1

cat $CURLOG >>$LOGFILE

DONE=$(grep ‘is not triggered’ $CURLOG | wc –l)

sleep 1

done

rm $CURLOG

log “End of import for jobId $jobId”

log “”

Now just put this in your /etc/cron.d/yoursite:

#
# Run product imports every morning at 6 am
#
0   6 * * *    yourwebsite    /home/yourwebsite/wpai_cron.sh -j 26
5   6 * * *    yourwebsite    /home/yourwebsite/wpai_cron.sh -j 27
10  6 * * *    yourwebsite    /home/yourwebsite/wpai_cron.sh -j 28

# Run product imports every morning at 6 am

0 6 * * * yourwebsite /home/yourwebsite/wpai_cron.sh –j 26

5 6 * * * yourwebsite /home/yourwebsite/wpai_cron.sh –j 27

10 6 * * * yourwebsite /home/yourwebsite/wpai_cron.sh –j 28

WooCommerce

Comments

18 responses to “WP All Import cron from CLI (update)”

February 7, 2015

lee

hi glad i found you, as after trying a cron on a large file i am getting the same problem of “last activity 5 hours ago”.

I wonder if you could help me a little, im a total noob?

I see where i must edit the xxYourKeyxx, but not sure about where to define the JOB (7?) number AND also hwo to check the path to the log? Currently logs for the wp al import when i trigger the cron are written to the

/public_html/wp-content/uploads/wpallimport/logs

—– my host told me

“/etc/cron.d/yoursite is the equivalent of the advanced mode editor accessible via the control pan”

so i guess i can add this via

/usr/bin/wget -O /dev/null -o /dev/null http://www.mydomain.com/wp-cron.php?import_key=MYKEY&import_id=7&action=processing

/usr/bin/wget -O /dev/null -o /dev/null http://www.wmydomain.com/wp-cron.php?import_key=MYKEY&import_id=7&action=trigger

help much appreciated, regards Lee

Reply
1. February 15, 2015
  
  rolandow
  
  Responded by e-mail. Lee’s webhost installed the shell script, problem solved :-)
  
  Reply
  1. May 23, 2016
    
    lee
    
    Yo roland can you help me out I need to use your cron again, but its stopped working if i give you shell access can you help me out, I trid the host, but is not helping. and access to backend wp can you check my setting for me. >?? can you email me an address to send the deets to if you have the time to take a look Cheeersss Lee :)
    
    Reply
March 17, 2015

Varun

Hi,

I ran the import cron to run every 2 mins and the trigger cron to run every 1 hour.

Now on the server root, the plugin seems to be creating countless files with the name wp-cron.php_import_key=[value]&import_id=12&action=processing.154….and so on….

Each file has the text:
Import #[number] is not triggered. Request skipped.
and is 54-53 bytes in size. Any idea what could be causing this?

Thanks much in advance.

Reply
1. March 17, 2015
  
  rolandow
  
  Seems to me that this is the output of the script. I’m not sure how your cron is configured, and if you are using some kind of control panel. You could try adding to your crontab rule: >/dev/null 2?&1. So the complete line would be:
  
  0 6 * * * yourwebsite /home/yourwebsite/wpai_cron.sh -j 26 >/dev/null 2>&1
  
  Reply
May 22, 2015

Dennis Bitsch

Hey,

Does your script still work?

I put it the script in the directory: /var/www/domain.com/
WordPress is in: /var/www/domain.com/htdocs/
Logs directory: /var/www/domain.com/logs/ (permissions are set right)

Also I did chmod +x on the script.
When I try to execute it though, it outputs nothing. No output at all, and it never ends. I modified it in one line only and changed public_html -> htdocs

Any ideas?

Reply
April 8, 2017

Daniel

Hi,

I have a lot of problems with my wp all import crons. After a while a cron shows the message “last activity….” and the whole site becomes really slow.

Is your script still valid? I don’t know coding, so I’m just trying to make the small changes like import key. :)

I created the file wpai_cron.sh with your code above and inserted my import key.

Then I have a cron like this
0 6 * * * /home/fanatiqu/wpai_cron.sh -j 46

However, it doesn’t work.

Do I have to create a logs directory and where do I put it? Anything else that I miss?

Your feedback on this would be highly appreciated since it causes a lot of problems.

Have a great day!

Best regards,
Daniel

Reply
1. April 10, 2017
  
  rolandow
  
  Hi Daniel,
  
  I don’t support the site anymore that was running this code. I am not sure if the script still works, and I am unable to test this. The shell script that I posted writes the output of the cron to a log file. So yes, you would need to create that directory or make sure you use one that can be written by the user that runs the cronjob. At line 26 and 27 you could change the log location to whatever location you would like.
  
  The output of the cron could give you hints why it’s not working.
  
  Good luck!
  
  Reply
2. December 29, 2017
  
  Hope
  
  Hi Daniel, did you ever find a fix for this? WP ALL Import cron jobs completely crashed my site and too am needing a solution.
  
  Roland have you ever considered creating a plugin for this? I wish WP ALL Import would. My guess is that anyone who uses cron jobs for stock updates is going to run into this problem. Any insight is much appreciated.
  
  Reply
  1. December 30, 2017
    
    rolandow
    
    Performance was really poor indeed. I remember I ended up writing my own script specifically for updating stock. This was not configurable at all but it did the job really fast.
    
    If you have a large product feed, then wp all import isn’t the best way to keep your stock updated. I think you’re better off if you develop a script on your own.
    
    Reply
February 23, 2018

Ben Besselink

Great Script! The only Problem is the speed. And i know it’s not because of your script. But i don’t know why.

I have a csv with 8500 products.

If i set the iterage to 20 and do the import via WP All Import Backend, it tooks only 1-2 hours.

If i set the iterage to nothing (empty) and set to high speed and do the import via cli, it tooks more than 24 hours.

dammed, where is the bottleneck?

Reply
1. February 26, 2018
  
  rolandow
  
  Try setting iterage as high as possible. If it’s too high, you will run out of memory. So I guess you could try 50, 100, etc. I think I’d stop at 100 if the time is acceptable then.
  
  I haven’t used WP All Import since that post anymore, but indeed, I found the plugin pretty slow as well. Especially if you just want to update the stock information. Eventually I wrote my own script that would just update the stock. If you create multiple update statements that execute at once, this can be really fast. I remember the script only took a few seconds, with thousands of products.
  
  I think the bottleneck is because WP All Import is pretty complex because you can configure so many options. It’s a really cool plugin if you cannot code yourself, although I imagine it can be hard to get it working then. But because of this, I don’t think it works as efficient as possible. If you write custom code for your specific feed, you can be a lot faster.
  
  Reply
April 20, 2018

Duc Sue

I am currently working on a site that requires daily updates to an xml file. This cron script looks ideal to what I want to do; however, I am confused (Newbie) to what this line below means. I am guessing the LOGFILE & CURLOG is the start dir from where ever the wpai_cron.sh is placed?

# Set magic variables for current FILE & DIR
__FILE__=”$(test -L “$0” && readlink “$0” || echo “$0″)”
__DIR__=”$(cd “$(dirname “${__FILE__}”)”; echo $(pwd);)”
LOGFILE=”${__DIR__}/logs/wpai_cron_${jobId}.log”
CURLOG=”${__DIR__}/logs/wpai_cron_${jobId}_current.log”
DONE=0

Can you also explain what does ‘website’ and ’26’ represent?
0 6 * * * yourwebsite /home/yourwebsite/wpai_cron.sh -j 26

I am assuming ’26’ is the jobID? What do I input as the ‘yourwebsite’?

Will this work?
0 6 * * * /home/www/wpai_cron.sh -j 26

Reply
1. April 20, 2018
  
  rolandow
  
  The LOGFILE and CURLOG point to files where the output of the cron is being written to. So this location needs to be writable to the crontab user. These logfiles are used to detect if the processing was complete. If you read the script, you see that it searches for the string is not triggered. That’s why it needs a current log, because in the large log this string will always be found.
  
  The cron definition was just an example. It depends on how your server is setup, but it needs to run under the same user as your webserver. The 26 is the job ID indeed. I’m not sure if your crontab definition works; I think it depends on where you define it. But I think there are enough articles on the internet that explain how define a crontab for your linux distro.
  
  Reply
August 11, 2018

Ryan

Hi,
I’ve got this up and running and it’s importing fine but there is a problem where the import never actually ends and untriggers. Any ideas of why this could be and any method of fixing it?

I’ve tried setting up the cron to hit the sh file every 2 mins / 30 mins / hour / 24 hours, none of these options actually end it. So it ends up in a constant cycle of importing.

I’m wondering if it’s due to the import rules I have set that tell it to not import if there is no image to the product being imported this results in the log files sticking at around 2500 of 4000 products imported as an example. I’m not sure what’s going on with it, any ideas?

This is in the logs with the cron set to 30mins:

{“status”:200,”message”:”Records Processed 2376. Records imported 2375 of 4214.”}Sat Aug 11 10:30:02 BST 2018: Start import for jobID 6
Sat Aug 11 11:00:03 BST 2018: Start import for jobID 6
Sat Aug 11 11:30:03 BST 2018: Start import for jobID 6

Reply
March 9, 2020

Mohammed

Hi,
Thanks for you for sharing the code.. Just wonder how could we fix error handling there. so for example if there is timeout and url does not work..
{“status”:500,”message”:[“cURL error 28: Connection timed out after 5001 milliseconds”]}

Reply
May 1, 2020

rolandow

I’m sorry, I can’t support this idea and script anymore. It’s been years since I wrote this post, and I don’t have this code working anymore. I guess you would have to dive in to bash scripting and see how you can process the result. Basically you could check for “status”:200 with grep I suppose, because anything else would be faulty.

Reply
May 1, 2020

Lee

Mohammed, If you work it out please post the result here. Cheers, Lee

Reply