Integrating Rollbar to Scrapy

Scrapy logs

For the past three (3) years, I’ve worked on multiple Scrapy projects with lots of spiders. Most of them are scheduled via scrapyd, a JSON API to schedule spiders. Sometimes, this spiders go kaput for various reasons such as change in layout, change in URL, being blacklist, among others. Checking the logs for them one-by-one can be time-consuming – so here’s where Rollbar comes in.

Rollbar is an error monitoring service that groups similar errors and gives you insights which one occurs the most. It can even help you track which commit/versin introduced the bugs. This way, you can discover bugs faster making it quicker for you to fix them.

Installing Rollbar for Python

Fortunately, there’s pyrollbar. You can install it via pip:

Integration pyrollbar to your Scrapy spider

Then, on your base spider (the spider that your rest of your spiders will extend, hook an instance of the RollbarHandler to the loggers of scrapy, twisted, and the spider itself.


Pink Eyes: Things to know about conjunctivitis

For a week now, I have been suffering from conjunctivitis, or more commonly known as pink eyes or red eyes. According to Wikipedia, “Conjunctivitis is the inflammation of the outermost layer of the white part of the eye and the inner surface of the eyelid.” It is called pink or red eye because it makes the eye appear pink or reddish.

Please don’t take this as medical advice. It’s just a digest of the things I know about conjunctivitis. Ask your doctor if in doubt.

Conjunctivitis, or pink eyes Continue reading “Pink Eyes: Things to know about conjunctivitis”

Side projects once more!

Ever since I started in my job at Scrapinghub, I realized that I stopped doing side projects – projects that I love to do on my free time. One of the main reasons I haven’t been doing them is poor time management. I get guilty doing side projects without completing my required number of hours at work.

Since the start of this week, my ex-Bright officemates who are now my colleagues at Scrapinghub started going to a co-working space near us. I noticed that I’ve been far more productive ever since thus eliminating the guilt to do side projects.

I’m still interested in doing the stuff I used to do before, such as maintaining UP student number databases for my personal consumption. My new interests include archiving or weather data via Project NOAH, archiving traffic data from MMDA and getting all sorts of potentially useful data. I’m interested in making analyses from those data and find interesting patterns from them.

Good luck to me and my new endeavor!

Poor service from HTC Philippines and their service center

I must admit that HTC smartphones are aesthetically-pleasing. The build quality is at par or better with the iPhone and obviously no match to the flagship phones of Samsung which looks pretty much the same and has the plastic feel. But before buying an HTC device, you might want to consider their poor aftersales support. Continue reading “Poor service from HTC Philippines and their service center”

Poor aftersales support from HTC Philippines and WSI PhoneCare

I have been an HTC fan ever since. I’ve had the first commercial Android phone released to the public, an HTC G1. I tried to get myself a Google Nexus One before but it was too much of a hassle so I got myself it’s twin, the HTC Desire, instead. My latest HTC phone was the HTC One.

I bought my HTC One last December when my phone that time died on me. As usual, I was very satisfied with the phone except for its disappointing camera. It did the job for me for my day-to-day tasks. It was the phone I brought with me to the US, good thing I bought an unlocked version. I used it to snap photos of my trip so most of my photos were there since I did not bring any camera with me. Continue reading “Poor aftersales support from HTC Philippines and WSI PhoneCare”

MRT3 website, hacked by AnonymousPH?

MRT3 has been all over the news lately, from the long queues, not enough trains to accommodate the riding public, the extortion scandal exposed by a diplomat – among others. Unfortunately, most, if not all of them are bad news.

That’s what made me think on what app to build to help the riding public. I tried to brainstorm. First thing to do was to do an inventory of data available to me. Unfortunately, it’s next to none. The only thing that I found was the CCTV live footage from the stations. Continue reading “MRT3 website, hacked by AnonymousPH?”

AlternaTeaser: TextTeaser rip-off powered by PyTeaser

I’m mostly a PHP guy, I admit it. But for the past few months, I’ve been diving into Python – mainly since I need to use it at work. Fortunately, Python isn’t the hardest language to learn.

AlternaTeaser is one of my attempts to be familiar for using Python for web. Basically, it’s a rip-off of TextTeaser – an article summarization awesomeness by Jolo, a former colleague of mine.

AlternaTeaser is powered by the Python port (the PyTeaser) of the original TextTeaser which was written in Scala. It also uses Flask framework for processing the web requests, nginx as the web server and MongoDB to keep all the metadata and summaries. For the frontend, I used Bootstrap and jQuery. Continue reading “AlternaTeaser: TextTeaser rip-off powered by PyTeaser”