Introduction
Have you ever wanted to buy that pretty dress or gadget really bad but disappointed at it being currently out of stock? Then, you check the listing once every few hours in hope to snap it up when it is available again. But what if the seller replenishes the stocks while you are busy or asleep due to time zone difference? One solution is to automate the checking of the product listing. We will explore simple shell scripting instead of the more advanced web scraping with Python for this tutorial.
Background
I had always wanted to buy a budget NAT VPS from Singapore but they are hard to find because data center rental is expensive and thus not many providers are willing sell it cheap. While KVMs are largely available, for example, in DigitalOcean but NAT VPS is not. I chanced upon this VPS provider who sells NAT VPS from Singapore but is currently out of stock. I opened a ticket to the provider and he replied that it will probably be be restocked by end of the month if there are no renewals. Thus, that made me check the listing quantity every day.
Shell Script to Automate Checking of Product Status
#!/bin/bash
if ! wget -qO- "https://clients.vpsprovider.net/cart.php?a=add&pid=13" | grep -q "Out of Stock"; then
#echo "available"
echo "https://clients.vpsprovider.net/cart.php?a=add&pid=13" | mail -s "VPS - NAT256 (SG) Available" -r sender@yourdomain.com recipient@gmail.com
fiCreate a text file and save it as check-outOfStock.sh. We use wget to get the product web page sources. Next, we grep to check for keywords – Out of Stock. The ! means to action on if keywords not found and this could be to send out an email to notify us that the product listing no longer contains the keywords.
(info) Sending Email from Command Line
# FAIL: does not have From email address
$ echo "testing message" | mail -s "message subject" youremail@gmail.com
# FAIL: bad address -f syntax from /var/log/mail.log
$ echo "testing message" | mail -s "message subject" youremail@gmail.com -- -f info@sgodds.com
# FAIL: bad address -r syntax from /var/log/mail.log
$ echo "testing message" | mail -s "message subject" youremail@gmail.com -- -r info@sgodds.com
# FAIL: returns mail: conflicting options
$ echo "testing message" | mail -s "message subject" youremail@gmail.com -f info@sgodds.com
# SUCCESS: but returns You have new mail in /var/mail/clindatc
$ echo "testing message" | mail -s "message subject" -r "Inception<info@sgodds.com>" youremail@gmail.com
# SUCCESS: PERFECT returns nothing at command line
$ echo "testing message" | mail -s "message subject" -r info@sgodds.com youremail@gmail.com
# SUCCESS: PERFECT returns nothing at command line
$ echo "testing message" | mail -s "message subject" -a "From: info@sgodds.com" youremail@gmail.comThe last two mail commands work and there are no differences. Both emails passed SPF and DKIM checks and do not generate errors in /var/log/mail.log during sending. I read that the mail options will fail dependent on Linux distro. In any case, the mail tests and shell script are tested on Debian 9 (stretch).
Adding Shell Script to Crontab
# m h dom mon dow command
*/5 * * * * /bin/sh check-outOfStock.shHow kiasu are you? ‘kaisu’ is a slang word in Singapore where it means the fear of losing out on something to others. For the cron job example above, a check will be conducted every five minutes and if you are asleep for seven hours, your mailbox will get spammed with (60 / 5) x 7 hours = 84 emails!
Conclusion
I was so excited that I jumped out of my sofa when I received the above emails. The provider had restocked ten quantity and I subscribed to one no. We can also modify the script to email on price change by changing grep -q “Out of Stock” to e.g. grep -q “$28.90” where $28.90 is the normal price. However, if you don’t own a VPS like me, a shared hosting plan will do fine as well. You can upload the shell script using FTP and easily set up cron jobs in cPanel or DirectAdmin. So now, let’s sit back, relax and our IT systems will automate the checking of price or quantity for us!