Aggregating data from multiple Craigslists with Yahoo Pipes

One of my computer programming specialties is data conversion, moving data from one format to another. I’ve been using Craigslist to hunt for jobs with “data conversion” in the listing. But there are hundreds of Craigslist locations, so I wanted to automate it.

(formerly Craigslist Helper) will search multiple Craigslists for keywords. It presents them one feed at a time. This is helpful, but I wanted a way to combine them.

yahoo pipes example

That’s where Yahoo Pipes comes in. This is a simple to use, drag and drop service from Yahoo, that among other things, allows you do pull data from multiple sources into a single RSS feed.

Here’s what I did to pull Craigslist “data conversion” jobs from multiple California sites into one feed.

  • Search for “data conversion” jobs. I started with Craigslist Los Angeles site
  • Click the RSS button on the screen that returns the search
  • Copy the url in the browser address bar into a text file. It should look like this:
  • Make as many new versions of that url as you need in the text file, changing cities for each one.
  • Go to
  • Click “Create a Pipe”
  • Click “Fetch Feed”
  • Click the plus sign next to URL in the Fetch Feed box. Add an url from the text file.
  • Repeat for all the urls.
  • Drag a connector line from the circle at the bottom of Fetch Feed to Pipe Output.
  • Click Save (and name it)
  • Click “Back to My Pipes”
  • Click “View Results” for the saved pipe.

Now comes the really powerful part

  • Click “Get as RSS” and add it to your RSS feed reader. You now have one RSS feed with search results from multiple Craigslist searches in it. I’m reading mine in Google Reader.

You can do this with any RSS feed. Yahoo Pipes allows you to pull into data from all sorts of sources, manipulate it, then output it in multiple ways.


  • Thanks for this post. I really appreciate it…

Powered by WordPress. Designed by WooThemes