Use own User-Agent

[hodduc]

Some sites (including https://news.ycombinator.com/) seems to deny http requests having "User-Agent: Python-urllib/.", causes add_feed() fails.

One simple proposal can be setting User-Agent: EarthReader/x.y OR User-Agent: Mozilla/5.0 before making requests.

(This should be applied on libearth.crawl (and any other modules using urllib2.urlopen) as well)

[tribela]

It might be fixed with earthreader/libearth#50.

[tribela]

Fixed in earthreader/libearth#50

[dahlia]

[11:42:50] <hodduc^talk> TheK-: https://github.com/earthreader/web/issues/44 요거 https://github.com/earthreader/web/blob/master/earthreader/web/app.py#L272 요기도 수정해야 하지 않을까요?
[11:43:22] <hodduc^talk> (T`heK-님이 진원님 맞으시..겠죠?)
[12:30:19] <hongminhee> hodduc^talk: 그러게요

[tribela]

Why not using crawler to that operation??
I think it should be change like crawler.check_reachable()