big torrent reaches100% but reverts back to 96% after hash-checking?

[LovesTheSun]

HI,

I think Bitcomet is great. It has very extensive features and detailed information regarding torrents. The online help is clear. It really shows there's been a great deal of work on both the client and the guide. Thank you all!

Now my question,

I have a big torrent that has 70GB of data. Last night when this torrent hit 100% my harddrive began rattling. I assume BC was hash-checking all the files in the torrent. When it stopped the torrent reverted back to 96.8% and started downloading again. Why did BC do that? I use 1.22 stable release

I must include I've selectively downloaded part of this torrent at first. When that job reached 100% with success. I ticked some more files within this torrent to be downloaded and let it run. But then the above mentioned thing happened. It is still running and I'll report back when it does this again. Chances are this will not occur again, I hope. But I was just wondering why this happened. Anyone got a clue?

Never experienced this with other torrents within this same version of BitComet I'm currently using.

TIA

Edited October 10, 2010 by LovesTheSun (see edit history)

[kluelos]

This usually happens when some part of the download fails the hash-check. Hashing is a very accurate way of making sure what's received matches what was sent. It's much more accurate than previous methods like checksums and CRC's, but does take longer.

BitComet will optionally recheck the hashes for downloads just completed, and if it finds a mismatch for any piece, discards that piece and attempts to download it again. This appears to be what's happening to you.

All downloads are susceptible to the real world, where connections are poor and telephone lines are noisy, and data gets corrupted. The bigger the download, the more errors can creep in, so the more likely this is to happen.

It's only when you get repeated hashfails on the same piece that corrective action is indicated.

[LovesTheSun]

I understand and thank you for explaining. Btw. it happened again. this is the second time. Should I be worried?. I wonder why so much data was corrupted. The torrent is currently redownloading 1 GB out of 25.3 GB. The first time the torrent had to redownload 995 MB so it seems to be growing? Any thoughts welcome

Three curious questions:

downloaded data is only hash-checked after downloading the whole file instead of on a piece-size level?

Can seeders modify the data on their hard-drive belonging to a certain torrent, ie. one picture, and my BC will download that file regardless and check later?

"It's only when you get repeated hashfails on the same piece that corrective action is indicated": Is there any way within BC for me to find out if this is the case? I made a screen-shot of the piece map now

Edited October 10, 2010 by LovesTheSun (see edit history)

[kluelos]

No, downloaded data is always hash-checked as each piece is completed. You can optionally have BitComet also run a hash-check on the whole torrent when it completes, to catch any errors that slipped through.

Modified files will no longer pass the hash check, so any piece that contains a portion of a modified file will be rejected.

I'm not sure where BitComet is keeping its hashfail count these days, but the behavior is the clue. You'll get to 99%, then stick. BitComet will download, then seem to be about to complete the download, but the piece will be rejected. BitCOmet will seem to be downloading at speed, yet stuck at 99%.

This can also be caused by deliberate sabotage, the intent being to have you waste time downloading a torrent which will never complete, in hopes of discouraging you. Suspect this if you are downloading a very recent popular movie or album.

[LovesTheSun]

No, downloaded data is always hash-checked as each piece is completed.

TY for your answer in the other sub-forum. You explain things very clearly.

By the way I might be having trouble here, to sum it all up here's what happened so far:

*I ran a torrent that downloads 70GB of data,

downloaded only part of the torrent because the torrent contains lots of files,

task completes 100% and hash checks OK.

*Then ticked some more files within this torrent to download and let it run,

task completes at 100% but after hash-checking reverts to 97.8% complete,

so I let it run and task completes at 100% but after hash-checking reverts to 96% complete (getting worse?)

*Task is running again and downloads to 98.5% and I stop it to run a manual hash-check out of curiosity,

and it ends up saying 95.5% (getting even more worse)

I don't get it, what could be going on here?

More infromation:

torrent health status varies between 300-1200%

total amount of data to complete: 25GB/79GB

no emule plugin enabled

1st retry: 955MB needed to be redownloaded

2nd retry: 1Gb needed to be redownloaded

currently: 1.113 GB needed to be redownloaded

TIA!

Edited October 11, 2010 by LovesTheSun (see edit history)

[kluelos]

I would select just a few files out of the torrent, a few hundred MB, setting all of the others to "do not download". Run the task and get those files.

Now move them to a different path.

Set the bittorrent task not to download that first set of files. then select a second small set of files and download those.

Repeat this until you've got all of it.

If you don't have the disk space for the whole thing, then make your download in chunks that make sense for burning off to an optical disk, then you can delete them from your hard drive.

If you keep getting errors, in a file set, try a different set.

In this process you may isolate the ultimate issue to just a few files or perhaps even one. That would certainly help in figuring out where the problem is.

[LovesTheSun]

That's a good idea!

If bitcomet loads a torrent-file for downloading, does BitComet modify this torrent-file in any way?

I thought i'd also to check if I could refetch the .torrent-file and compare it 1:1 with the existing version to see if there's something wrong.

And one more thing, it;s a big torrent of 80GB, whenever I open BC and click on that torrent, it takes forever before any menu will respond. Also if I want the properties for the torrent, it takes ages for any button to respond. do you perhaps know why that is?

Thanks Kluelos!

Edited October 11, 2010 by LovesTheSun (see edit history)

[kluelos]

No, BitComet doesn't modify the .torrent file. It does copy the file to a working directory, and also to its archive, and renames the archive copy to the torrent's hash value (for lookup purposes).

There shouldn't be a delay for getting torrent properties. The cause of any is elsewhere in the system -- it's busy doing something else, or there's not enough memory or the like. You should notice that while the torrent material may be 80 GB, the .torrent file itself is tiny and doesn't put an appreciable load on the system.

[LovesTheSun]

And what about maybe the piece_part.bc! construction within BC? I'm not attacking BC in any way I'm just trying to rule out possibilities.

If I read correctly, more data is downloaded to single files, if the file-length is smaller or bigger than piece-size boundaries. Could this in anyway have something to do with what's happening?

If not it will leave me to my Hard-drive as culprit.. (download SMART utility to check)

or there's lots of errors due to connectivity issues from peers like you said at first

funny thing is, I looked up some files where bitcomet says it is incomplete, and when I open them they appear to be fine to the eye (pictures though)

Also one time I downloaded 20some files, deleted the piece_part.bc! file in directory and rehash-checked. then BC said it had to download 7% of the torrent again. but before I let the torrent run I copied the supposedly incomplete files, and when the supposedly incomplete files completed, I compared them 1:1 with the copied files, 100% same. strange?

Thank you for your time, your very patient!

Edited October 11, 2010 by LovesTheSun (see edit history)

[kluelos]

A typical picture contains much more data than your display can show, or your eye can see.

Your eye can't actually see and discern anything like sixteen million distinct colors, but the data can be there to display them.

Since you can't actually SEE this data, it's possible to make a picture file smaller by not including all of it. A compression algorithm that discards some of the detail data is called "lossy" as in "loss of data".

Pictures can be quite incomplete then, and still appear to be acceptable.

For your homework (don't worry, this is fun), look up and explore "steganography"

The incompleteness question is answered in the other thread.

[LovesTheSun]

Kluelos, do you know why bitcomet is reacting terribly slow with big torrents in the tasklist (80GB lots of smal files)

Starting bitcomet, clicking other torrents, their options, changing save location entering trackers is all normal.

But when I click the big torrent, bitcomet will hangfor 40sec. Rightclicking on torrent will hang for 40secs, clicking options will do the same,

clicking tabs to go to advanced settings will waste you 1.5 minutes, tabs, fields will seem non-responsive bc it takes 50secs before it will respond.

Sometimes they wont respons at all unless I repeatedly click on e button that seemingly doesn't work (non-clickable)

Also downloading this big torrent, BC will stop downloading after 10 minutes and become non-responsive, don;t know what is happening, then after a minute it will resume downloading. But needs to build up speed again and the torrent availability is recalculated. It does this repeatedlyafter 9-10 minutes. Could it be BC is writing cache data to files, HD activity led is burning when this occurs but disk seems awfully quiet..?

Tia

Edited October 12, 2010 by LovesTheSun (see edit history)

[kluelos]

THere's nothing I know of about BitComet that would cause this slowness. Nevertheless, why not try an earlier version, like 1.18 and see if the slowdown still occurs?

Version 1.17 changed storage locations so downgrading past that version gets tricky. Doable, just tricky. You can download every version that's ever been released, from the main bitcomet site.

The most likely cause though, is something to do with your system.

Bittorrent stresses a system in unusual ways. Not bad ways, just ways that most other applications don't. Because of that, it often provokes system problems that it gets blamed for, when it's actually a victim of them.

The next step in weeding things out is to temporarily try another client and see if it has similar issues.

[LovesTheSun]

Hi Kluelos!

I'm downloading now and about what I mentioned earler, about the torrent freezing every ten minutes for 40-5-secs. it doesnt do that anymore.

well it does but only for 1 sec, and sys-monitor says it's bc of the harddrive writing data to disk.

I suspect my anti-virus, it has a feature which filters incoming web-data through various protocols like http. I Disabled it and after two restarts it finally greyed out/meaning it is disabled.

Could anti-virus software that uses "filtering of imcoming protocol-streams" mangle the data?

I downloaded about 1 gig while it still halted every ten minutes so I'm paranoid it might have. or should hash check at the end rule any corrupt data out?

And where can i find the copy of the torrent within BC that BC uses to work with. I wanna redownload the same torrent and compare if it still has the same hshh values for all the data

(yeah i know, I'm super paranoid)

[kluelos]

Most antivirus software waits until the data has been written to disk, then actually accessed by anything else.

Trying to step into the middle of the stream to read it sounds like a way to cause trouble. It also involves mucking with the system on a level best not mucked with. In order to intercept the stream you need to make a lot of assumptions about things that are subject to change without notice to anyone-- because nobody's supposed to be fooling around in there. There's no reason to think that this makes the antivirus product more effective, so I would pass it by.

Another hash check won't hurt anything, but I do think you're going well beyond what's reasonably necessary.

Find the torrent copies in %appdata%/BitComet/Torrents

[LovesTheSun]

I'm getting tired over here figuring out what the ... is going on..

IE. when only 4% data seemed corrupted after downloading/hashcheck which BC needs to redownload, each time BC retries to download the missing pieces this total amount of corrupted data, will decrease one might think. Because BC retries and finds all sources until the torrent reaches a healthy 100%.

But This is not the case with my torrent. 97% complete with 3% bad data will that needs to be redownloaded, increases to 4% bad data after a retry. To make it more confusing, my torrent went from 656MB to 506MB to 424MB data that needed to redownloaded, to 690MB after the last retry.

My question is , what can cause this to to happen?

[greywizard]

This is not a very common issue so there is no straight easy answer to it.

Sometimes, trying to download a torrent in which a user inadvertently included temporary files from his system may cause this behavior. But we have no way of telling if this is your case.

If the files are always the same, and they are not important, you may try to disable them (set them not to download) to see if your torrent reaches and stays at 100%.

Sometimes, there are ill-conceived torrents (uploaded by anti-piracy organizations shills) which are meant to never reach 100% as they keep feeding corrupted data into the swarm (using hacked clients) in order to hinder downloading for everyone who tries to download that torrent.

But there is no easy telling if that's your case either.

However, have you managed to isolate the files which need re-downloading?

Are they always the same, or they change after each hash-check (check the Files tab to see the percentage of each file)?

If they are the same, then what type of files are they?

[LovesTheSun]

Yes I have, and this is what I see:

The torrent consists of many pictures and soundfiles. Let's say the picture directory is sorted like this "70-125 pictures per directory" Not all directories have been included for download. When I isolate the pictures that always fail to reach 100% this is what I see: it is always the beginning 3-4 and last 4 pictures (800-1MB per file) in a directory that are corrupted. Looking at the torrent saying the piece-size is 4MB I think there might be a link. But what exactly I don't know.

Instead of of selectively downloading part of the torrent I decided to download everything, bc the BC_part file might have something to with it. I can at least try.

[bobgreen]

I have too this problem after upgrade my BitComet up to ver. 1.22. Many tasks which downloaded 100% after some time some tasks show 99.8% complete and start redownload this 0.02% again. In previous versions I've never seen it before.

Edited November 4, 2010 by bobgreen (see edit history)

[vatsav]

I am using 1.36 stable release and I too have this problem.

A set of files were downloaded properly and hash checking was also done successfully. Without disturbing this set, I started adding new set by checking few more files. When this set gets downloaded and hash check reached 100%, percentage actually goes back to 96% disturbing the first set as well.

Now, even files in first set will have some percentages like 90+. Lot of bandwidth wastage..! I've been using bitcomet from years but never faced this issue till now. I am talking about a 120gb file which I am downloading in parts(sets).

These files are important to me and I have no idea on how to proceed. :( :( :(

Edited July 6, 2014 by vatsav (see edit history)

[The UnUsual Suspect]

This is not a problem with BitComet, this is happening because you are using bittorrent protocol in a way it was never intended to be used. The option to select files within a torrent wasn't part of the original protocol and if you change the files selected in the torrent the result is you have to download some pieces of data again in order to construct those files.

I'll also add that torrents won't download as efficiently if you disable files and generally it's only recommended to do this if you never intend to download those files. If you're disabling files because you want others first, it would be better to just set the priority on the ones you want higher so they download first.

Imagine you have a torrent with two videos in it and there are a total of 9 pieces of data between them, to download the first video you need pieces 1-5 and for the second you need pieces 5-9. Piece 5 is shared between the two and if you then enable video two you'll have to redownload piece 5 because half of it was discarded as unneeded, so video one reverts to incomplete.

BitComet did introduce a way to avoid this by using padding files to assure that no piece contained data from two files, but the rest of the clients and their users objected so strongly that we turned off this option, but you can still use it when making torrents however some trackers won't allow the torrent simply because they don't understand it and don't care to learn about the benefits.

The simple solution is to use a torrent the way it was designed to be used and if you absolutely must change the selected files after download is complete, then you'll have to accept the fact that you're going to have to redownload some data.