large commit size

[bc65536]

Hi, BitComet on my machine sometimes uses more memory than I think is appropriate and commit size grows over time rather than reaching steady state. (E.g. Over 1 GB commit with 512 MB physical.) Is this a known problem? I did not see other posts or notes in the FAQ about this magnitude of memory usage. Are debug symbols available to help identify the cause?

Sometimes BitComet.exe will consume less than 200 MB after running for hours, but sometimes it grows excessively. For example today after about 12 hours the VM size as shown in Task Manager is 970708 K. It is not a leak because when exiting cleanly the commit size comes down to 32 MB at the point kernel32!ExitProcess is called. The current BitComet rate is roughly download 2 kB/s, upload 85 kB/s.

Versions:

OS: Windows XP (Version 5.1 build 2600.xpsp_sp3_gdr.100216-1514 : Service Pack 3)

Physical memory: 512 MB

BitComet: 1.22.

eMule plugIn: 1.22.6.8

Router: NetGear WPN824v2

BitComet settings:

Global max download rate: 9999 kB/s

Global max upload rate: 80 kB/s

Listen port: set to same as value in router, Statistics pane shows port as opened

Enable UPnP port mapping: on

Enable Torrent Exchange: on

eMule plugin: Installed, enabled

Appearance/Default task info tab: statistics (and tab for "start page" is not visible)

Shrink disk cache size if free physical memory is lower than: 80 MB

Thanks

[The UnUsual Suspect]

I personally haven't seen any similar reports about 1.22, and I'd say unless your running an extreme number of tasks, this is highly unusual.

For question I can think of. Is this behavior new to version 1.22? or have you experienced it in prior versions?

One thing I have seen is when an excessive number of stopped tasks are uploading on LTseed, like 100 or more, it can use excessive amounts of memory and many users don't understand LTseeding and expect bitcomet to be idle when tasks are stopped. As a test, you can simply disable this option and see if the excessive memory usage stops. If it does, then you just have to better control how many tasks you allow to LTseed.

If this is something new since installing 1.22, reverting back to a previous version would be an option. It's impossible to test all new versions on every possible system configuration, connection, and usage, so some incompatibilities will surface from time to time.

[bc65536]

Your advice worked great! After many hours the commit size is only about 56 MB. I turned off "enable long term seeding".

For LTSeed there seems to be only a rate control, with units kB/sec. Is there a way to directly control number of LTSeed tasks? Next I can try limiting network.max_connections and enabling LTSeed.

Thanks!

[greywizard]

All the tasks which have LT-Seeding support (i.e. non-private and for which a LTS hash has been calculated by any BitComet peer) will be available for LT-Seeding upload.

That is, any task which is finished and either stopped or seeding and fulfills the above conditions will be available for LT-Seeding. Therefore, deleting tasks which have reached the target share ratio from the Task List, may be a solution.

If you really need to keep all the present finished tasks in the Task List, you can disable LT-Seeding on a per-task basis, from the properties page of each task.

[The UnUsual Suspect]

Just so your understanding the purpose and behavior of LTseeding. This is a proprietary network run by Bitcomet where your completed tasks are shared with other users. Enabling this option will allow you to also download from other bitcomet users who have this option enabled. BitComet is unique in the industry because of it's ability to download from a variety of sources. Your public torrent can be downloaded via traditional bittorrent peers, http/ftp servers, emule or LTseeds, or any combination of these.

After your completed tasks have fulfilled their seeding ratio to bittorrent peers, and if your bandwidth isn't needed for existing active tasks, the stopped tasks will begin to upload to LTseed peers.

In your case, I'm assuming that you have a large number of stopped tasks that are in this category, which is growing the amount of resources that your system needs to seed all these files, so by simply better managing the stopped tasks, either by removing some, or turning off LTseed (on selected tasks), you can greatly reduce the amount of memory bitcomet needs.

TuuS

ps. I'm sorry for the typos in my previous post.

[Doctor_D]

I'm having the same problem as the OP, using BitComet 1.23

I'm using a quad core with 2GB ram and XP Pro SP3 (unmodified tcpip.sys)

BitComet starts off using about 60MB of memory and ends up using about 1.2GB after 24 hours.

LT seeding is off.

I'm downloading 3 torrents & uploading 3.

[terrykorkor]

same here, ram usage(ram + virtual memory) is large, especially when you are downloading at quite fast speed, and is significantly increasing at more than 2mb/s.

when downloading at around 7-9mb/s, the "cache" seems to be unable to handle them all and throw them to virtual memory and in the end i can chunk up to 1.5-2gb of virtual memory usage. And when i stop the task, bitcomet will say disk busy, which i assume it is slowly copying the data to the hdd.

i'm downloading just 1 task, at 300 global task, 100 max simultaneous half open tcp, using version 1.20 for now, and i'm thinking if 1.23 would help(though from a few report over here in the forum, it seems this problem still persist in 1.17-1.23)

[kluelos]

when downloading at around 7-9mb/s, the "cache" seems to be unable to handle them all and throw them to virtual memory

Well, that isn't true. Virtual memory is written to/read from the HD, and simply is not used in this way. It's use is *entirely* under the control of the OS, and not at all under the control of the application. Bitcomet, or any other application, CANNOT "throw them to virtual memory." Nor is the casual user able to determine, much less control, at any given moment what virtual memory is being used for or how much of it is being used.

And when i stop the task, bitcomet will say disk busy, which i assume it is slowly copying the data to the hdd.

That is true and is expected. At some point the cache must be written to disk.

The price for minimizing disk writes is that each write takes longer, because you are writing a lot of data in one chunk instead of small amounts of data in several chunks. Was this not understood?

i'm downloading just 1 task, at 300 global task, 100 max simultaneous half open tcp, using version 1.20 for now, and i'm thinking if 1.23 would help(though from a few report over here in the forum, it seems this problem still persist in 1.17-1.23)

No problem's been established yet, far less existing in other versions. You have yet to describe one. Neither did the original poster.

Memory exists to be used. That use does not, of itself, constitute any problem at all.

If it is in some sense "better" that the memory sit there not being used, please tell me how you think that's better?

[Doctor_D]

This is actually a significant problem.

I agree that a program should make use of extra memory, if available, to enhance it's performance.

However, it should also relinquish that extra memory if other processes require it.

Bitcomet 1.23 is not doing that.

It uses more and more memory, until there is none left and eventually it crashes.

I noticed the huge increase in memory use as soon as I upgraded from v0.94 to v1.23

(As stated previously - I was only downloading 3 torrents & uploading 3.)

CRASHLOG.TXT follows:

BitComet caused a Microsoft C++ Exception (0xe06d7363)

in module kernel32.dll at 001b:7c812afb.

Exception handler called in Exception_Debuger.

Error occurred at 12/8/2010 11:35:02.

C:\Program Files\BitComet\BitComet.exe, run by Doctor_D.

Operating system: Windows XP (5.1.2600).

4 processor(s), type 586.

94% memory in use.

2047 MBytes physical memory.

115 MBytes physical memory free.

4096 MBytes paging file.

4096 MBytes paging file free.

2048 MBytes user address space.

140 MBytes user address space free.

Program name: BitComet

Process execute time: 2 day 3:23:14

Program status: Running

Program mode: download

Socket init version: v2.2

web frame:

msg id: 275

piece cache size: 52428800

bt mem_block_size: 52428800

crash info:

debug info:

pool_block dump:

allocated_size = 4390912

allocated_list_size = 67

free_block_size = 1339392

free_block_num = 327

commited_size = 3051520

notblock_size = 52428800

notblock_number = 30

total_size = 56819712

Context:

EDI: 0x00000000 ESI: 0x0012f1d4 EAX: 0x0012f14c

EBX: 0x00000000 ECX: 0x00000000 EDX: 0x00cd5cb8

EIP: 0x7c812afb EBP: 0x0012f19c SegCs: 0x0000001b

EFlags: 0x00200206 ESP: 0x0012f148 SegSs: 0x00000023

Bytes at CS:EIP:

5e c9 c2 10 00 85 ff 0f 8e 36 93 ff ff 8b 55 fc

Stack: [ignore]

Module List: [ignore]

===== [end of CRASHLOG.TXT] =====

[kluelos]

Six simultaneous tasks is more than most connections can support if they're all active. You may want to check again.

When you run too many tasks, they divide your available bandwidth among themselves. It's your upstream bandwidth that is so limited, and if you starve a task for bandwidth it begins slowing down dramatically. This is because you are a slow and unreliable trade partner, so the fastest connections in the swarm tend to find faster ones than you to swap with.

As a result you're left with only those connections that couldn't find anyone better to connect to themselves. Every peer tends to find its own level, and all connections are mutual decisions. You get only those connections that everybody faster has already rejected.

--------------------

Meantime though, you should first understand that the error log is of no help in this case. The exception is a very general "Microsoft C threw an exception" and is used for any error that is raised by the MSVC++ compiler through a call to "throw". Note the code, e06d7363. It's "E" for exception, followed by M-S-C, in ascii. I'm afraid that's useless. It indicates nothing about what went wrong or where, simply that something did.

There is no evidence that this was associated with memory usage at all. In particular there is no indication that the problem is caused by running out of memory. The log says that 94% of memory is in use, leaving 6% left over, so the preliminary appearance is that being out-of-memory is not the issue.

There's no evidence that it's not either, of course, but prejudging what went wrong is an effective way of keeping yourself from seeing true causes and solutions. The default cache size is 50 megabytes, which is what the log shows your system was using.

The current version is 1.24. Do you still see these problems with the current version?

0.94 was released over three years ago. Is there a reason you delayed upgrading for so long?

We are not seeing widespread reports of constantly growing memory usage in many systems, which you'd expect with a significant memory leak.

That suggests that something else, something about your system or software in particular, may be the root cause. You should review the software that you typically run, and try disabling things to see if the problem slows down or goes away.

[Doctor_D]

Six simultaneous tasks is more than most connections can support if they're all active.

They weren't giant swarms with hundreds of peers. Total download speed was about 125KB/s and upload was 65KB/s (limited by me).

This has nothing to do with bandwidth saturation or excessive number of connections. Even with only one active task (upload only ~ 20KB/s) memory consumption continues to climb.

There is no evidence that this was associated with memory usage at all. In particular there is no indication that the problem is caused by running out of memory. The log says that 94% of memory is in use, leaving 6% left over, so the preliminary appearance is that being out-of-memory is not the issue.

I really don't know how you can draw that conlusion. If I start an application that requires more than 120MB of memory while Bitcomet is using 1900Mb, then I will run out of physical memory. Bitcomet doesn't seem to 'play nice' under these circumstances. My system grinds to a halt while my drives get a heavy workout paging stuff (everything except Bitcomet stuff, it seems). It's a memory hog - no application should use 94%+ of physical memory in a multi-tasking environment.

The default cache size is 50 megabytes, which is what the log shows your system was using.

Then what's Bitcomet using the other 1850MB for exactly?

The current version is 1.24. Do you still see these problems with the current version?

I haven't upgraded to 1.24 yet, as I can see others are having the same problem with it.

0.94 was released over three years ago. Is there a reason you delayed upgrading for so long?

Never had a problem with 0.94, so I didn't bother to upgrade for ages. 0.94 never used more than 450MB of ram, even with 16 active tasks going flat-out.

We are not seeing widespread reports of constantly growing memory usage in many systems, which you'd expect with a significant memory leak.

That doesn't mean there isn't some kind of problem present. I've noticed that this isn't the only thread referring to this issue.

You should review the software that you typically run, and try disabling things to see if the problem slows down or goes away.

Agreed. I'll do some testing over the weekend.

[kluelos]

Look at the crashlog with me.

First, what is there, that to you, indicates the system is out of memory?

It says, "94% of memory in use". I don't see a problem there, not all by itself. I need to see something more than just memory being used, some evidence of intensive action in the swapfile. I'm not saying that isn't occurring, I'm saying this log doesn't show that. I would expect about that much memory to be in use given your OS and total installed memory as a stable condition.

It is you, who have jumped to the entirely unwarranted assumption that BitComet is using that 94%. The log simply does not say that. It says only the total memory in use, not who's using how much of it.

XP will run, albeit very slowly, in 1 GB of RAM, but that's really not enough, XP doesn't really wake up until it has at least 2 GB to work with. That's what you've got. I would expect most of it to be in use most of the time. If you give it another gig of ram, I would expect most of THAT to be in use most of the time. And if you added 2 more gigs, THAT would be used. What I would never expect is that there would be large blocks of memory that are never used. WinXP will run in 2 GB. WinXP will run in 4 GB. You are probably swapping less in 4 GB than 2 GB, but you are not swapping excessively in 2 just because it is 2 and not 4.

I don't think it's excessive and I don't see evidence that this, all by itself, indicates a problem. Memory is there to be used. Swap files are there to be swapped. That by itself isn't a problem, it's normal for a system to page. It's only when we're actually running in swap that there gets to be a problem. SO what does your crash log say about it?

Does it say that your paging file is full? Or does it say that the paging file is completely free and (at least at the moment of the crash) not being used at all?

4096 MBytes paging file.

4096 MBytes paging file free.

The log also says the piece cache size and the memory block size are at 50 MB, where they're supposed to be. But I don't see anything in this log that indicates that *BitComet* is taking excessive amounts of memory. Now, possibly, it is, but this log doesn't say so, and it would certainly be unwarranted to assume that's the case without evidence.

*You* may have seen excessive memory usage, but *this log* doesn't show it.

It seems to me that you've decided for yourself how memory allocation must work, in a way that doesn't reflect reality -- that if an application ever allocates any memory, it's never given back until the application terminates. That just isn't so. This isnt' the place for a course in how Windows allocates and recovers memory, though.

Do you have some evidence that this behavior you're worried about is actually occurring, that memory is being locked up forever?

I'm not saying that you don't have a problem, but I am suggesting that you are so focused on pre-judging that BitComet is the cause of it, that perhaps you aren't looking at evidence suggesting BitComet may simply be another victim of an entirely different problem. But I suspect you would agree that, if you were only looking at the crash log, you'd wonder what the guy was wittering on about? While we'd like to help you find the solution, we can't do much if you insist on telling us things that the log plainly shows, just aren't so.