Showmedo caught some flak from the recent Westhost (our main server-provider) troubles. Catch this pretty interesting thread for the gory details. We were actually pretty lucky as the main site remained standing (not the case for many others) but did manage to lose our cron daemon. As a result some restart scripts which check on the health of, among other things, the Python Turbogears process running the site died. Now this script isn’t needed too often <fingers crossed> but if the site falls down and we’re literally asleep then it avoids more than a minute or sos downtime. The crontab line in question looks like this:
*/1 * * * * source /.bashrc && /usr/local/bin/python2.5 /home/showmedo/showmedo/cron_start_showmedo.py > /var/log/cron_log
This line tells cron to run cron_start_showmedo.py every minute and the python process will check the health of Showmedo’s main process, our memcache and mysql servers and a few other things and restart them if anything has fallen over. Having lost cron, I needed a quick hack to perform these duties and found some very cool advice here at Slashdot. For various reasons the recommended bash-scripts seemed to be dying on me, and lacking the time or will to try debugging on fairly foreign turf, I hacked a little Python substitute to do much the same:
1 import time 2 import subprocess 3 4 while 1: 5 subprocess.Popen(['/usr/local/bin/python2.5', '/home/showmedo/showmedo/cron_start_showmedo.py']) 6 time.sleep(60)
Not pretty, but at five lines quick and succinct. The line 4 while loop sets things running in perpetuity, line 5 uses Python’s newish (and pretty powerful) subprocess module call the cron_start_showmedo.py module and line 6 takes a 60 second break before continuing the loop.
The script sat duty while I waited for our cron-daemon to return and I slept sounder in my bed as a result.If you enjoyed this post, make sure you subscribe to my RSS feed!