I know it's an old topic but since I had this problem until 5 minutes ago I'll post here my solution anyway.
Kluelos' work-around is not an option because by deleting the preferences file you actually trash all the settings you've made on your Firefox addons. So I found far more simple way to fix the problem without deleting any files. First I set the browser to use BC to open magnets and torrents. Of course there was this error message, so I did this:
• Set BC to be default app for magnets and torrents (there's a drop down menu which contains the words "application by default" in rounded brackets; select this).
• Open BC and remove the checks from checkboxes about default torrent client and about magnet URI. Click APPLY, then OK. No need to close the settings dialog window.
• Put the checks back on the checkboxes about torrents and magnets. Click APPLY again, then OK.
Try to open a magnet or a torrent link. It will work.
Basically what I did whas to force BitComet refresh its settings and it worked. And no files were hurt in the process. :D