Log in

A smart way to read NEP reports - Живые заметки Фёдора Исхакова

> Recent Entries
> Archive
> Friends
> Profile
> My Website

August 18th, 2010

Previous Entry Share Next Entry
09:59 am - A smart way to read NEP reports
How do you keep track of the new papers in your field?

Here is the description of what I think is a smart way to do it.  I spent some time developing this multi-step solution after trying numerous other ways which worked but did not quite give the right feeling.  I'm writing this note to check back later whether this solution is smart enough, and also hope that this code may help some other people.

What you need: http server with enabled PHP. This setup also uses NEP reports from RePEc database, Google reader and JabRef reference manager. The scripts are written in PHP and Python. I run this setup on Ubuntu, but installing this on Windows seems also possible.

Step 1. I find the RePEc database fascinating. Among other services connected to it, there is NEP - frequently updated reports on new papers in different fields of economics. Recently they introduced RSS feeds in which each entry is a single paper.  I think this is way more handy than getting email reports to your inbox.

Step 2. I think Google reader is a wonderful RSS reading software, especially because of the one feature that facilitates the whole thing. Subscribe to the NEP RSS feeds in a usual way by copying the RSS link from the report page or clicking on the RSS button in the browser while on the report page.

Google reader allows to have a custom "Send to" link accessible in [Settings]/[Send to] tab. This provides a way for a particular paper in the NEP report to be consumed by the custom PHP script that prepares the entries for the BibTeX file, which can be further processed by a Python script run locally to download PDFs of the papers and store them so that JabRef reference manager knows where they are.\

The user experience is the following. Follow the RSS feeds with new papers from the NEP reports you subscribe to in Google reader.  Feed items will contain title, authors and usually an abstract. When the paper is worth reading, click the appropriate link to "send" it to the PHP script. In response you instantly get a link to the RePEc page of the paper, and the paper is taken into the list of new reads.  Such monitoring of the new papers can be done on any mobile device connected to the internet, thus you can utilize, for example, the commute time in the morning. Once at the office, run the local script to download the saved papers in both BibTeX and PDF formats to your workstation.  JabRef will display the correct links to the PDF files enabling easy browsing, and all the new papers are instantly ready for referencing.

Step 3. Download the needed scripts.

Step4. Both PHP scripts and the JabRef icon should be uploaded to preferably a separate directory on your server, say <myserver.com/newread>.  Test that the scripts are actually working by directing your browser to http://<myserver.com/newread>: the page you see should have blank NEP, Title and URL fields and the RePEc link should point to RePEc homepage.

Step 5. Configure Google reader by setting the custom "Send to" fields to:
  Name = Your name for the "Send to" option, for example, "Add to my reading list"
  URL = http://<myserver.com/newread>?source=${source}&title=${title}&url=${url}
  Icon URL = http://<myserver.com/newread>/jabref.png
Test by sending any paper from NEP feed to your reading list.  This should result in a similar page as in the previous step, but with the fields filled out and the box containing BibTeX file of the paper.  Note that BibTeX entry has a "file" field which points to the link provided in the NEP report.  Most of the times this is a link to PDF and it is therefore treated as such, but occasionally the link leads to another html page, in which case PDF will not be downloaded automatically.  You have to download it manually from the corresponding RePEc page.

Step 6. On you local machine run the Python script get_newread.py in a folder which already has pdf/ subfolder in it for PDFs.  The script updates the bib file newread.bib with the new papers, downloading information from <myserver.com/newread>.  The script has to be edited before use to provide the correct address to the server.
File links are reformatted to the JabRef style (I'm not sure whether it is universal) and point to pdf/ folder where PDFs are downloaded to.
Personally I run this script in the folder with all bib files each of which referencing PDFs in the pdf/ subfolder. This way copying entries between different bib files in JabRef preserves the links to PDFs.

Step 7. After the script is finished, open newread.bib in JabRef to browse the papers.

UPD: After realizing that the php script can work just off the title, I introduced two additional tools.
When you call newread.php without any arguments, a fields appears to enter the paper title manually. The updated code is downloadable from here: ver. 1.1
In Google Chrome it is possible to have an extension that allows for the submission of selected text on the page - with a special new item in the context menu. The code is available from here, and the installation instructions is found here in list item 4.
Current Location: Sydney

(1 comment | Leave a comment)


[User Picture]
Date:August 18th, 2010 11:41 am (UTC)
Cпасибо! Очень полезно!!!!

> Go to Top