Yahoo killing off Delicious.

17 December 2010

It was announced by Yahoo a few days ago that, amidst layoffs they will be shutting down a number of popular services, among them the online bookmark repository-cum-social networking site Delicious.

Some people are asking why any of us would bother sharing our collections of bookmarks online; about the only thing I can tell you is that Delicious (formerly made it possible to access our bookmarks from any system on the Net and not just the computer they're saved on.

As you might imagine this causes something of a problem for those of us who fall into that particular use case. Somebody at Hazelnut Consulting came up with a way to migrate your bookmarks to an open source package which does the same thing called Scuttle. The article states that you should use a service called Delicious XML to download the contents of your account in the form of an XML file which can then be imported into an instance of Scuttle to pick up where you left off. There is a slight problem with this, however: if you were a use of Delicious before Yahoo bought them out, Delicious XML will not work because it's hardcoded to use your Yahoo account to download your bookmarks, and if you predate that time your Delicious account won't be hooked to your Yahoo account. I tried this a few times this morning but found my efforts stymied by this limitation. But there is a solution.

When Yahoo bought out Delicious, they did not take the API server (which is how you interact with the service's back end) offline, they simply extended it. It's still there, still usable, and here's the documentation which was used to work up this example for downloading all of your bookmarks (with tags and associated metadata) into an XML file suitable for importation elsewhere. I ran into a problem on Dreamhost this morning when setting up Scuttle, though: if you have too many bookmarks you'll get server errors during the import because the file is too big. I found a way around that when looking at the options for the 'posts' command. It is possible to download your bookmark collection in chunks, save each chunk to a separate file, and then import them one at a time to restore your collection. Here's how I did it:

curl -A "Mozilla/4.0 - Virtual Adept Delicious Bookmarks Downloader" -u virtadpt -o bookmarks-000.xml\?\&results=200

In my example, bookmarks-000.xml is the first chunk of my bookmark collection. The \?\&results=200 bit of the URL I loaded with the curl utility (available as a package for just about every Linux distro I know of) told the API server to send me the first 200 bookmarks from my collection. The -o option of curl sets the filename to write to. The &'s and ?'s are backslash-escaped because your shell will attempt to process those special characters otherwise, and then the command won't do what you want it to. Once I'd verified that it worked I then moved on to subsequent blocks:

curl -A "Mozilla/4.0 - Virtual Adept Delicious Bookmarks Downloader" -u virtadpt -o bookmarks-200.xml\?\&results=200\&start=200

Here I've changed the filename to write bookmarks to (bookmarks-200.xml) and instructed the API server to send me another 200 bookmarks starting 200 places deeper into my collection (\&start=200). I did this over and over, incrementing the numerical value in the output filename and the offset into my collection by 200 each time until I'd downloaded my entire bookmark collection. From there it was a simple matter to import each block of URLs into my new collection. I did this by clicking the 'add a bookmark' link, clicking 'import bookmarks from' at the very bottom of the page, selecting a bookmarks-*.xml file, and clicking the 'import' button.

The -A option sets a custom user-agent string sent to the web server on the other side; the Delicious API docs ask you to do so out of politeness. The -u option of curl sends your Delicious username to the API server. When prompted, type in your Delicious password. Do not forget to increment both numbers, otherwise you'll overwrite the last file you created!

You can probably do this with wget if you've a mind to, but I stuck with the example I had for expediency.

UPDATE (17 December 2010 @ 1414 EST5EDT): Official word from Yahoo about the fate of Delicious.