Windows 2008 R2 - large file copy uses all available memory

[guest]

I have a problem that was discussed in the following link but never resolved. I'm unable to reply to that thread, so I've created a new one in the hope that someone might be able to help.

http://social.technet.microsoft.com/for ... f=required

I have two Windows 2008 R2 servers, and I'm trying to copy large (minimum 50GB) files back and forth between the servers. If I copy a 50GB file from server 0 to server 1, the transfer rate stays at just below 1 gigabit/sec on a gigabit switch. However, if I copy a 50GB file from server 1 to server 0, the copy begins at just below 1 gigabit/sec, but once the amount of data transferred is equal to the amount of available RAM on server 0, the transfer rate steadily decreases (will continue to decrease rapidly and might level off at just 50 megabit/sec). It doesn't matter if the file is pushed or pulled.

Server 0 is a Dell PE2950 with 24GB of RAM and 2 dual core Xeon 5110 CPUs @ 1.6GHz

Server 1 is a Dell PE2950 with 32GB of RAM and 1 quad core Xeon E5420 CPU @ 2.5GHZ

I have seen this happen before on Windows 2008 x64 without R2, and I've used DynCache http://www.microsoft.com/downloads/en/d ... laylang=en to resolve it. However, DynCache is not supported on Windows 2008 R2, and it's not supposed to be needed on R2 because the problem was supposedly fixed / solved. Interestingly, I only have the issue on one of the two R2 servers.

In task manager on the problem server, as soon as I start the file transfer, I can watch the available memory begin to drop. At the moment I have 24GB of RAM in the server, and about 16GB of that is available. Once 16GB of the 50GB file has been transferred, the available memory gets down to 0 in task manager, and then the transfer rate tanks. The OS was installed just a week or two ago. It has Hyper-V and SNMP installed, as well as the latest Windows updates. I then installed the File Services role as well, but the problem still exists. Nothing else has been installed.

Clearly there is still an issue here in Windows 2008 R2, but it doesn't seem to affect all servers in all situations. There are also clearly other people having the same problem, but to my knowledge Microsoft has yet to acknowledge or address the issue in Windows 2008 R2. Can anyone help?

[guest]

I would suggest to check if this issue persist with perform the commands below to disable the TCP Chimney Offload/Receive Side Scaling feature in Windows server 2008 R2.



netsh interface tcp set global rss=disabled

netsh interface tcp set global autotuninglevel=disabled

Reboot the server



For background information, please refer to the article below:



Information about the TCP Chimney Offload, Receive Side Scaling, and Network Direct Memory Access features in Windows Server 2008

http://support.microsoft.com/kb/951037

[guest]

Thank you for your response, Tiger Li. I followed your instructions and did the following, but the problem still exists.


netsh interface tcp set global rss=disabled

netsh interface tcp set global autotuninglevel=disabled

Reboot the server




Any other suggestions?

[guest]

After all this time, I finally "solved" the problem I was having. It seems like in this posting there are possibly multiple issues being discussed, because it's not clear that everyone's situation is that same as mine, so keep that in mind when you read what I changed to make the problem go away in our environment.

In summary, it all came down to the write-caching policy on the RAID controller. We are dealing with Dell servers and Dell controllers, and I have reproduced the issue on both embedded/internal RAID controllers as well as their external RAID controllers for direct-attached storage arrays.

When the RAID5 virtual disk on the controller is set to write-through, the copy performance issue exists. When the virtual disk is switched to write-back, the problem disappears. By default we always use write-back caching, but when the RAID battery fails (or while the battery is charging), the controller automatically switches the virtual disk back to write-through until the battery is replaced or charged. You can select to "force write-back" when the battery is dead, but the consequence is possible data loss if the server were to crash or lose power during a write operation.

Interestingly, the performance difference of write-through vs write-back caching often seems negligible. However, for certain operations, including large file copies across a network, there are clearly issues. Interestingly, I have not been able to reproduce the problem when copying files on the same server from one array to another. It only seems to exist when copying across a network where the destination drive has a write-through caching policy.

I hope this info helps some other people with similar problems. I can't believe that after all this time and so much testing that it all came down to a single setting. It's disappointing to discover that the write-caching policy can have such a large impact for some operations and a nearly non-existent impact for others.