SMB v3 file integrity issues

Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware
Goto page 1, 2, 3, 4  Next
Author Message
CanadaMoose
DD-WRT User


Joined: 12 Apr 2017
Posts: 50

PostPosted: Wed Dec 16, 2020 16:47    Post subject: SMB v3 file integrity issues Reply with quote
Using R7000 running DD-WRT v3.0-r44863 std (11/24/20), NAS with a ext4-formatted drive connected via USB.

I've been experiencing file integrity issues when transferring large files (100MB+) from Windows 10 after moving to the R7000 (from WNDR3700v2 on older version, which did not support SMB v3). The files got silently corrupted (no errors thrown), so it took me a while to figure this out.

After much investigation, it seems that SMB v3 is at least partially responsible. I use a powerline network, which may also have contributed. I did not see any issues transferring files using Wifi. Setting the max version to SMB v2.10 seems to have resolved the issue for me.

I strongly recommend to enable Encryption in the NAS settings, which also enables integrity checks. At least you'll get feedback that the transfer failed. Enabling this did not significant increase CPU load on my device.
Sponsor
CanadaMoose
DD-WRT User


Joined: 12 Apr 2017
Posts: 50

PostPosted: Mon Dec 28, 2020 20:00    Post subject: Reply with quote
Got a long cat6 cable to eliminate powerline ethernet as the culprit. The issue is still happening on wired ethernet. Wireless connections are fine.
Also found that it does not seem to be related to the protocol version as uploaded files (to NAS drive) get corrupted with SMBv3 and SMBv2. The integrity checks only seem to be working with SMBv3, though. NAS throughput is about 20-30 Mbps on a 1Gbps network connection, but the CPU is not taxed (maybe 20% busy).
Anyone with similar issues?
CanadaMoose
DD-WRT User


Joined: 12 Apr 2017
Posts: 50

PostPosted: Thu Dec 31, 2020 3:34    Post subject: Reply with quote
Swapped out the drive enclosure, but the issue remains: Copying large files from Windows 10 fails eventually. Next, I will try the copy from another device / Ubuntu to rule out my setup and Windows 10.

Does anyone know the samba version currently included in the latest builds? Tried to find it, but ...
EDIT: Looking at the source, seems to be ksmbd v3.1.9
lexridge
DD-WRT User


Joined: 07 Jun 2006
Posts: 363
Location: WV, USA

PostPosted: Fri Jan 01, 2021 4:48    Post subject: Reply with quote
Maybe try reverting versions of dd-wrt. r44715(r44719) is a good one apparently. It's worth a shot anyway.
_________________
Linksys EA8500 (Internet Gateway, AP) - DD-WRT r44719
Netgear R6250 (AP, VAP) - DD-WRT r44772
Linksys EA8500 (Gateway Backup) - DD-WRT r44236
Features in use: Samba, OpenVPN, WireGuard, Entware for Mosquitto
Member #248
CanadaMoose
DD-WRT User


Joined: 12 Apr 2017
Posts: 50

PostPosted: Sat Jan 02, 2021 20:28    Post subject: Reply with quote
Tried copies from Ubuntu and Windows 10 from different devices. Downgraded to r44700, all same result, file corrupted. Three findings, though:
- Copies using wifi network complete successfully. This leads me to believe that samba is not the culprit
- First 1GB file after reboot completes
- There are no CRC errors showing in the ifconfig output.

Will try a few more things:
- Go way back when I have some time, something like r41459 that was running on my WNDRv2 without issues
- Reset to factory and only setup minimum required for the samba access

Will report back.
flood404
DD-WRT User


Joined: 18 Dec 2013
Posts: 234

PostPosted: Sun Jan 03, 2021 18:10    Post subject: Reply with quote
I had trouble with windows 10 on my best computer here with a Ryzen 3900x. The older systems running the same version of 10 worked fine. For some weird reason I could not copy files over. I could not play any mp3 or video file without getting a EOF error or invalid argument.

Did not matter which player I used. Linux Mint worked fine of course on the same problematic computer.

Then I found out. If I set a password and user name but did not check mark the option for samba access in dd-wrt. The Windows 10 computer I have issues with would ask for user name and password anyways. I could just type in one random character in each field and it would let me access the shares. Even if the password and user name does not exist in the dd-wrt samba configuration it still lets me in. I just do not use the remember the user name and password option on Windows 10 and get prompted every time I reboot the Windows 10 pc. I prefer to do that since it cured the corrupted files, EOF errors and invalid arguments.
CanadaMoose
DD-WRT User


Joined: 12 Apr 2017
Posts: 50

PostPosted: Sun Jan 03, 2021 22:38    Post subject: Reply with quote
Quote:
Even if the password and user name does not exist in the dd-wrt samba configuration it still lets me in.

Just tried that and access was cut off, as it should be. You describe very strange behavior and may want to try to factory reset and rebuild of the router config.
CanadaMoose
DD-WRT User


Joined: 12 Apr 2017
Posts: 50

PostPosted: Mon Jan 04, 2021 23:11    Post subject: Reply with quote
Build r44700. Saved my config, reset to factory using "erase nvram && reboot" from command line. Disabled wifi and setup USB support and NAS. Large file copy working, no issues.

Restored my config. Large file copy still working. Will monitor for a few days, then upgrade to latest build.
howard30
DD-WRT Novice


Joined: 07 Mar 2014
Posts: 33

PostPosted: Tue Jan 05, 2021 22:31    Post subject: Reply with quote
CanadaMoose wrote:
Build r44700. Saved my config, reset to factory using "erase nvram && reboot" from command line. Disabled wifi and setup USB support and NAS. Large file copy working, no issues.

Restored my config. Large file copy still working. Will monitor for a few days, then upgrade to latest build.


I have R8000 and I have no luck with all newest firmwares: all kinds of issues. With 44715, it seems working out OK. Throughout rates are about 20MB/s, up/down.
CanadaMoose
DD-WRT User


Joined: 12 Apr 2017
Posts: 50

PostPosted: Tue Jan 05, 2021 23:07    Post subject: Reply with quote
As bushant wrote here https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=327735 the correct command for builds after 35770 is: nvram erase && reboot

Seems I got lucky that the older command (applied in error) solved my issue. Anyway, I don't think there is any harm in trying.
sierratango
DD-WRT Novice


Joined: 22 Jul 2014
Posts: 48

PostPosted: Fri Jan 08, 2021 4:45    Post subject: Reply with quote
Thanks for bringing this issue up as it may never have caught my attention. I have experienced the same file integrity issues as you mentioned but my situation seems a little worse. Transfers over Wi-Fi also get corrupted and large transfers over SMB 3 not only get corrupted but often overload the router (DIR-868L) causing it to reboot.

I'm seeing the issue in the latest build (r45229) and also the builds mentioned in this thread (r44700 and r44715). I did perform resets too. However, downgrading the builds and using the SMB 2 protocol seem to result in fewer corruptions.

I wrote a small shell script to generate an arbitrary number of random data files of any size to send over the network to check for corruption using md5sum. Hope it's useful for testing this issue.
howard30
DD-WRT Novice


Joined: 07 Mar 2014
Posts: 33

PostPosted: Fri Jan 08, 2021 16:00    Post subject: Reply with quote
sierratango wrote:
Thanks for bringing this issue up as it may never have caught my attention. I have experienced the same file integrity issues as you mentioned but my situation seems a little worse. Transfers over Wi-Fi also get corrupted and large transfers over SMB 3 not only get corrupted but often overload the router (DIR-868L) causing it to reboot.

I'm seeing the issue in the latest build (r45229) and also the builds mentioned in this thread (r44700 and r44715). I did perform resets too. However, downgrading the builds and using the SMB 2 protocol seem to result in fewer corruptions.

I wrote a small shell script to generate an arbitrary number of random data files of any size to send over the network to check for corruption using md5sum. Hope it's useful for testing this issue.


Thanks for the script. I didn't do any md5sum checking with r44715 as I only checked copying stability. Since r44715 has many issues with 5Ghz channels, I have downgraded to Kong's r40270M. I used your script to test 10 100MB file size and they all passed. Will try again with r44715.
egc
DD-WRT Guru


Joined: 18 Mar 2014
Posts: 7444
Location: Netherlands

PostPosted: Fri Jan 08, 2021 16:35    Post subject: Reply with quote
A long and winding thread, perhaps some background information might help Smile

In the beginning there was Samba3, not very fast but with a reasonable memory footprint then came Samba 4 which was even slower and had a huge memory footprint so not really suited for little routers.

But then came the solution a new kernel samba was developed fast, low memory footprint: ksmbd
Huray, one tiny problem it is still a Work in Progress (to put it mildly) at this moment (and we are now using 3.3.1) it is usable for your regular USB stick if you use ext 2/4. For NTFS there still could be problems.

I think the last build of 2019 (yes more than a year ago) still uses the old samba3 after that it was ksmbd (somebody has to check this as I am not sure).

The last update to ksmbd was with build 44971 so although build 44715 has stable wifi it is not the latest and greatest for ksmbd (not saying it will work in 44917 and after Sad )

Bottom line do what I do, I use a QNAP 453 Pro with 4*8TB Smile

_________________
Routers:Netgear R7800, R6400v1, R6400v2, Linksys EA8500, EA6900 (XvortexCFE), E2000 (converted WRT320N), WRT54GS v1.
WireGuard Documents & Guides:https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=327397
OpenVPN Documents & Guides: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=327398
IPSET: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=327261
Install guide R6400v2:http://forum.dd-wrt.com/phpBB2/viewtopic.php?t=316399
Install guide R7800: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=320614
Forum Guide Lines (important read):https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=324087
CanadaMoose
DD-WRT User


Joined: 12 Apr 2017
Posts: 50

PostPosted: Sun Jan 10, 2021 0:32    Post subject: Reply with quote
Well, I celebrated too soon. My problem came back after about 20GB transferred. Not sure yet why, but the Samba version might have something to do with it, after all. Version 3.2.3 (3/July/2020) includes
- fix stuck issue while writing files with windows client

Not sure if this is related to the issue, but it could be. I'm glad you guys chimed in, now I know I'm not alone and have a possible hardware issue.

I'm in the process of moving my r7000 to a test bed, so I can try different setups quickly.
@sierratango: The MD5 script will be helpful.

It looks like brainslayer is already working on updating the Samba version. Let me know if I can help test / verify something.
sierratango
DD-WRT Novice


Joined: 22 Jul 2014
Posts: 48

PostPosted: Wed Jan 13, 2021 2:39    Post subject: Reply with quote
After testing several older builds (with resets) that includes the in-kernel Samba server, I am confident the file corruption issue stems from the new implementation, at least for my particular router. Builds using the old Samba implementation do not have any corruption according to my tests. The last build to include the old Samba is r41986.

Below are my test results using an ext4 HDD.

New Samba
---------
r45229:
SMB 2.0 - 100 MB - 100/100 passed
SMB 2.0 - 500 MB - 0/1 passed
SMB 2.1 - 100 MB - 87/100 passed
SMB 2.1 - 500 MB - 20/20 passed
SMB 3.1.1 - 100 MB - 17/20 passed

r44715:
SMB 3.1.1 - 100 MB - 70/73 passed - crashed

r44700:
SMB 2.0 - 100 MB - 229/230 passed
SMB 2.0 - 1000 MB - 0/1 passed
SMB 2.1 - 100 MB - 296/300 passed
SMB 2.1 - 500 MB - 4/8 passed
SMB 2.1 - 1000 MB - 5/5 passed
SMB 2.1 - 2000 MB - 5/5 passed
SMB 2.1 - 3000 MB - 5/5 passed
SMB 3.1.1 - 500 MB - 1/5 passed - crashed

r44627:
SMB 2.0 - 100 MB - 6/6 passed - crashed
SMB 2.0 - 1000 MB - 0/0 passed - crashed
SMB 3.1.1 - 1000 MB - 0/1 passed - crashed

r42132:
SMB 2.0 - 100 MB - 98/100 passed
SMB 2.0 - 1000 MB - 5/5 passed
SMB 3.0.2 - 1000 MB - 4/5 passed
SMB 3.1.1 - 100 MB - 398/400 passed
SMB 3.1.1 - 500 MB - 10/10 passed
SMB 3.1.1 - 1000 MB - 10/10 passed
SMB 3.1.1 - 2000 MB - 4/5 passed

r42054:
SMB 3.1.1 - 2000 MB - 4/5 passed

Old Samba
---------
r41986:
SMB 2.0 - 100 MB - 400/400 passed
SMB 2.0 - 1000 MB - 20/20 passed
SMB 2.0 - 2000 MB - 10/10 passed

r41954:
SMB 2.0 - 100 MB - 100/100 passed
SMB 2.0 - 1000 MB - 5/5 passed
SMB 2.0 - 2000 MB - 5/5 passed

r38253:
SMB 2.0 - 100 MB - 100/100 passed
Goto page 1, 2, 3, 4  Next Display posts from previous:    Page 1 of 4
Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum