Testing WD Red 4TB SMR vs CMR HDDs in a ZFS NAS Array - Skip SMR

  Рет қаралды 73,793

ServeTheHome

ServeTheHome

4 жыл бұрын

Check out Will's main site article here: www.servethehome.com/wd-red-s...
We tested the WD Red WD40EFAX drive that uses DM-SMR or SMR technology instead of CMR technology in a ZFS RAIDZ array. We did some basic performance tests to confirm the steady-state operation. We then took a 4-drive array of CMR drives, failed one drive out of the array, and timed the rebuild process using three CMR drives including the CMR "EFRX" WD Red 4TB model and then the SMR "EFAX" drive. The results were so bad and so scary for a potential user, that we cannot recommend that people purchase these drives.
We discuss how iXsystems seems to have confirmed a similar, but different scenario/ issue and is de-certifying the drives for ZFS and FreeNAS use.
We also show how the lack of clear disclosure on just how bad this can get is dangerous for those who do not follow tech trends religiously (e.g. most NAS buyers in this segment.) Spread the word to avoid these drives and make sure to get the CMR variants instead.
Other articles referenced in this video:
- Surreptitiously Swapping SMR into Hard Drives Must End: www.servethehome.com/surrepti...
- Recovering from a recent Proxmox VE double-disk failure: www.servethehome.com/9-step-c...
- ASRock Rack X470D4U Review: www.servethehome.com/asrock-r...

Пікірлер: 370
@junkmauler
@junkmauler 4 жыл бұрын
The SMR Red's need to be rebadged as WD Browns ;)
@MiGujack3
@MiGujack3 4 жыл бұрын
Ah yes the WD Brown SMR drives (Shitty Magnetic Recording)
@TechyBen
@TechyBen 4 жыл бұрын
@@MiGujack3 Nothing wrong with SMR. Just like nothing wrong with a bicycle. Advertising it as a *motorbike* is wrong.
@MiGujack3
@MiGujack3 4 жыл бұрын
​@@TechyBen I was just making a joke about the brown branding. But it is shitty for NAS though.
@3dr14ng4
@3dr14ng4 4 жыл бұрын
Poopy brown.
@paulsim7589
@paulsim7589 4 жыл бұрын
@@MiGujack3 another word that fits is Stool, both another word for shit, and something to sit whilst you wait for the rebuild :p
@AndreasTyrosvoutis
@AndreasTyrosvoutis 4 жыл бұрын
Until WD removes all SMR drives from their "RED NAS DRIVE" line, I will not take them seriously, nor purchase WD any longer.
@IIGrayfoxII
@IIGrayfoxII 4 жыл бұрын
Same. I recommended WD, I hated Seagates. But I will now be using seagate for my NAS neeeds. Sure i am just a home user and not business, I do not want shit like SMR for my NAS.
@azuwan
@azuwan 4 жыл бұрын
This makes me think about how many WD Red with SMR drives in the market, if today they stopped SMR drive in WD Red, what about the drive is still in the market? I should have not to consider WD Red product line altogether. The least they can do is create a new RAID specific product family as WD Red family ruins by now. This fiasco will be remembered when WD Red is no longer marketed as for server applications(NAS=server; people generally regard the NAS specific product line with server application and thought NAS must be RAID supported drive). If I have important data on my RAID server I wouldn’t jeopardised using WD altogether for risking multiple drive failure which rendered unrecoverable data lost.
@IIGrayfoxII
@IIGrayfoxII 4 жыл бұрын
@@azuwan Making another line is stupid when the Red is made for NAS. doesnt matter if the NAS uses Ext4, ZFS or what ever as the filesystem. A NAS is a NAS. They ditched the green because people were complaining about them being "Slow" They were meant to be slow, they used less power, they made less heat. For backups or HTPCs these drives were fine.
@cataria3903
@cataria3903 4 жыл бұрын
u can chose your devil. seagate submarined SMR drives for years (just not nas) and also produce drives, that are less reliable. all hdd companies are shit and scammers. btw this also isn't the first time, that WD pulls out a middle finger to customers. look at the endless load/unload cycle madness, that infected tons of drives. drives literally parked their heads so much, that they dead much earlier than they should. just remember, that they are all shit, but it is very understandable to avoid any future WD products, after they deliberately destroyed data from users with "nas" smr drives.
@LostPhoenixGaming
@LostPhoenixGaming 4 жыл бұрын
All my NAS drives are CMR WD Reds but I wouldn't risk buying a Red now. I've had bad things with Seagate in the past hence why I'm all WD but I am actually considering the Ironwolf line given the assurance by Seagate that they will be CMR. The sad thing is, I buy a 4TB WD RED, it will be SMR but if I buy a 4TB WD Blue, it will be CMR. WTF? SMR on Blues I understand but in Reds? It a good way to kill your product line
@TacticalFluke09
@TacticalFluke09 4 жыл бұрын
Shit man, 15 times longer for a resilver is wildly, wildly problematic. Not acceptable. Excellent work STH
@pilotavery
@pilotavery 2 жыл бұрын
Considering it holds twice as much data on the same platter with this technology, I would be okay with it taking twice as long. That being said, you're right, 15 times is too much. Three times longer is too much. There are now some optimizations for SMR drives in raid arrays but even then it goes from 15 times to about six times, still absolutely ridiculous
@davidtsang1807
@davidtsang1807 4 жыл бұрын
Ever since in 2015 I bought my first SMR Ext. drive from Seagate, I was aware of the hanging/jerking behavior. So I totally shocked by when WD had made their RED drives to be SMR. They should discontinue the lower-end model of RED drives instead of using SMR to make profits or just label them as BLUE drive whatsoever. Labeling a SMR drive as NAS drive is a criminal.
@azuwan
@azuwan 4 жыл бұрын
WD should create new product line specific for RAID. After this fiasco I wouldn’t touch ANY WD Red, knowing that WD mess up the entire high end specific for server application product line with inferior products. Going forward, WD should more carefully in product lineup and more transparent.
@lloydmilton
@lloydmilton 4 жыл бұрын
​@@azuwan definitely, for server use if the SMR drives are an issue with NON ZFS systems then dun touch them at all.... From my perspective, the main question I have is it only affecting ZFS setups - which I understand is a lot of freenas users or is it much wider at an enterprise data storage level???
@Morpheus-pt3wq
@Morpheus-pt3wq 4 жыл бұрын
This may be a way to force customers interested in better HDD´s to pay more for "pro" or "gold" line.
@ShaferHart
@ShaferHart 10 ай бұрын
@@azuwan doesn't justify it but reds are not their server lineup, it's their nas lineup. Their enterprise drives are for servers, ultrastar etc.
@rdwatson
@rdwatson 4 жыл бұрын
Only 5 minutes in and this sounds damning. WD can ignore random people on the Internet but not STH.
@esra_erimez
@esra_erimez 4 жыл бұрын
This!
@CriticoolHit
@CriticoolHit 4 жыл бұрын
I mean.. They can... And they will. Sadly.
@davidg4512
@davidg4512 4 жыл бұрын
I consider the bait and switch a total scam. What shady tactics and full of corporate greed.
@unijabnx2000
@unijabnx2000 4 жыл бұрын
and also they didnt even reduce the price of the SMR drives they silently put in the place of the CMR drives.
@garym1550
@garym1550 4 жыл бұрын
fool me once, shame on me. fool me twice shame on you. This is not new corporate thinking, it's been around a long time. Only shaming the offending parties in public and affecting their bottom line by cutting their sales!, will bring them to the realization that they need to make a correction. The new people who make these poor decisions are also those who have not witnessed similar event in the past or simply say, "I don't care" and "I can get away with it. I have the power to do as I see fit." The question now becomes, do WE have what it takes to force them to see what they are doing is wrong and make them pay for their arrogance.
@cataria3903
@cataria3903 4 жыл бұрын
u know the really sad part? western digital expected this kind of backlash probably. maybe a bit less, but still. lawsuits are cheap compared to gaining tons of profit over a longer period, if they manage to somehow get SMR drives into nas setups. and WD also knows, that their competition is hated just as much, so they can behave however they want, in the end there are only 3 choices and all are shit. personally i am just shucking mybook 10 TB + drives myself now. why is shucking a thing? because the hdd prices of internal drives are also a total scam :D because of course they are.
@amp888
@amp888 4 жыл бұрын
This really feels like an example of false advertising on WD's part. If you're going to segment your product stack and advertise particular drives for specific use cases then you should ensure the drives are suitable for them. I feel like WD should offer a refund or exchange program for anyone who unknowingly purchased an SMR drive that was advertised for use in a NAS or RAID setup.
@DjRavix
@DjRavix 4 жыл бұрын
I actually bought those drives before it was public knowledge that they were SMR drives And had problems with them WD support provided me a RMA for them ... they are replacing my drives for Red Pro drives without me having to pay extra for those
@thorn3576
@thorn3576 4 жыл бұрын
@@DjRavix They probably knew that your problems most likely came from SMR snd send you a CMR drive so you would not get suspicious when the replacement was also faulty.
@Mr_Meowingtons
@Mr_Meowingtons 4 жыл бұрын
@@DjRavix i wish i had done that.... I got 4 to add to the 4 drives i had for a large RAID 6 stack. I just sent them back to the vender. All 4 where shitting them selfs during the rebuild..
@deth3021
@deth3021 4 жыл бұрын
Sounds more like fraud.
@BillLambert
@BillLambert 4 жыл бұрын
Smells like class action in the making
@paulsim7589
@paulsim7589 4 жыл бұрын
Yea there is one now. I'll let you google it. (two in fact, US & Canada)
@miketel01
@miketel01 4 жыл бұрын
I agree
@markhaus
@markhaus 4 жыл бұрын
I'm surprised the EU wasn't the first to do this
@that_llama_in_a_tuxedo4584
@that_llama_in_a_tuxedo4584 4 жыл бұрын
Marcus Grant you forget Americans are the most litigious
@mightyhalo
@mightyhalo 4 жыл бұрын
A friend of mine just almost bought one of those 4TB SMR drives for his Synology 5 bay NAS. I have stopped him in time and made him order an IronWolf instead. So glad I did that. The rebuild of Synology's 16TB formatted mdraid was finished after about 12 hours (a night basically).
@AllanSavolainen
@AllanSavolainen 4 жыл бұрын
MD-devices seem to work just fine on SMR, some test show them even to be slighly faster than previous generation CMR REDs :) ZFS is the problematic one and that can be mitigateed by using separate CACHE and LOG devices.
@chaos.corner
@chaos.corner 4 жыл бұрын
I've been NAS shopping and was looking at a double 4TB buy. I'm glad this came up. I almost pulled the trigger a few days before it did.
@chaos.corner
@chaos.corner 4 жыл бұрын
@@AllanSavolainen Presumably this is for LVM mirroring also? Think I'll still pass though.
@theshuff
@theshuff 3 жыл бұрын
What a big sleeper ;)
@FlyingToilet
@FlyingToilet 4 жыл бұрын
WD also has a 2.5" 2TB "Black" drive that uses SMR. Black is supposed to be the performance brand.
@TechyBen
@TechyBen 4 жыл бұрын
No problem if it's not advertised as NAS compatible without telling the NAS it's SMR.
@s1mph0ny
@s1mph0ny 4 жыл бұрын
@@TechyBen Nah, Blacks are meant as a cheaper alternative to an SSD where performance is really important.
@paulsim7589
@paulsim7589 4 жыл бұрын
I never noticed that...... they used SMR in a 'black' drive.... Eww. What were the management thinking.
@scottyhaines4226
@scottyhaines4226 4 жыл бұрын
@@s1mph0ny WD Black HDDs are supposed to be top of the line HDDs. Every HDD can be a cheaper alternative to SSDs.
@happygimp0
@happygimp0 4 жыл бұрын
@@scottyhaines4226 I checked a online shop for my country: cheapest SSD 32.50, cheapest HDD 40.90
@doryiii
@doryiii 4 жыл бұрын
9 days to resilver! And that's ZFS, which only rebuilds the 60% of filled data! @@
@AndrewFremantle
@AndrewFremantle 4 жыл бұрын
The issue is that ZFS doesn't resilver by disk area - it doesn't work in straight lines. It rebuilds in the order in which ZFS originally wrote the data, which means on an active array with changing files it's seeking all over the place. This was acceptable [if less than ideal] for CMR drives, but SMR drives just completely shit the bed in that workload.
@-morrow
@-morrow 4 жыл бұрын
@@AndrewFremantle yeah, ZFS would even optimise for SMR, but those drives don't advertise themselves as such to the host.
@heh2k
@heh2k 2 жыл бұрын
@@barn2255 It's zfs recreating the data/coding onto a new drive after replacing a failed one. It's similar to raid rebuild, except it's at the FS level instead of the block level. Pros: only needs to rebuild used space, Cons: it's not purely sequential like a classic raid rebuild.
@TimSedlmeyer
@TimSedlmeyer 4 жыл бұрын
WD are not idiots regarding HDD technology. They had to understand the issues customers would encounter with these drives and that the use of SMR wouldn't stay secret long. This along with their customer unfriendly blog statement which attempts to gaslight customers and shift the blame to them makes me believe that WD made the decision willing to accept whatever blow back occurred. They must really be seeing some manufacturing cost savings on these things. What I don't know if they considered is that the small NAS segment includes many who run a small NAS at home and enterprise NASs at work. And that those who do are also the ones most likely to cheap abreast of news such as this. This will negatively impact their enterprise business. They are already on my do not buy list.
@tommihommi1
@tommihommi1 4 жыл бұрын
the engineers at WD must be constantly facepalming at this decision
@TechyBen
@TechyBen 4 жыл бұрын
I've heard the manufacturing costs are like 60 to 75% less, as they can reduce the number of platters and read/write heads massively (4-5 tracks overlapping equals a 1/4 or 1/5th the number of platters!).
@cataria3903
@cataria3903 4 жыл бұрын
yes they knw, of course they knew, that this backlash would happen. also the increased profit is not only in scaming people by paying the same price for a cheaper to produce drive managed SMR drive, BUT also by pushing people into more expensive drives, that are still CMR. turning CMR into a "premium" technology, instead of the standard. the bs blog post even tells u, that u should buy those WD overpriced gold garbage drives, that will make u want to tear your ears out, if i remember correctly. so they want to take away your silent nas drives, sell u insanely loud "premium" drives and also on top of that one has to remember, that it was WD and seagate, who removed AAM (automatic acoustic management), which gave the user control about how loud a harddrive would be. it is just insanely. evil insanity.
@LostPhoenixGaming
@LostPhoenixGaming 4 жыл бұрын
its one thing to understand and another thing to care. an engineer can shout as loud as they like about the tech but if the finance department sees a big cost saving then the engineers' opinion would count for nothing. Remember, its managers and money men that run things and not the engineers or technical people.
@paulsim7589
@paulsim7589 4 жыл бұрын
I am sure they cost of legal action (ie class actions) and such will be vastly less than the money they made. However, I am not sure they have accounted that for the application RED drives are used in, people have long memory's and are the least forgiving in tech.
@ARandomOWL
@ARandomOWL 4 жыл бұрын
I specifically ordered some EFRX and received EFAX anyway. Returned them and went with a different brand.
@artlessknave
@artlessknave 4 жыл бұрын
yup, that's why they did it, because they expect most people won't notice....until AFTER their array dies anyway.
@Morpheus-pt3wq
@Morpheus-pt3wq 4 жыл бұрын
@@artlessknave customers will probably find out much sooner, as rebuilding raid with SMR drive takes ages.
@brianh.000
@brianh.000 4 жыл бұрын
Who did you order from? I'm looking at NewEgg for an EFRX.
@artlessknave
@artlessknave 4 жыл бұрын
@@Morpheus-pt3wq thats what i mean by their array dying, taking so long to rebuild that more disks die
@RN1441
@RN1441 4 жыл бұрын
WD needs to recall the SMR red drives, or justifiably we need to advise as many customers and friends as possible to avoid their products.
@denniskorner921
@denniskorner921 4 жыл бұрын
Good to know. But the video could've been 8 minutes without all those repititions again and again and again... 😉
@sjobbefin
@sjobbefin 4 жыл бұрын
Cleaning lady knocked the power off several times :)
@djvincon
@djvincon 4 жыл бұрын
Right arrow key skips 10 seconds.
@moebius2k103
@moebius2k103 4 жыл бұрын
It’s out of control huh.
@nickwallette6201
@nickwallette6201 4 жыл бұрын
This video’s script uses SMR.
@RootSwitch
@RootSwitch 4 жыл бұрын
Really appreciate your data and coverage of this. I was recently buying some cold spare WD Red 3TB models right before a lot of videos about the issue popped up. I was completely unaware of the problem. Just by chance the 2 used drives and new drive I got off eBay happen to be the older CMR models matching the 9 I have in production. Being "surprised" by a multi-day rebuild when your array is already degraded doesn't sound fun. Eventually I'll phase these out, but I'll be looking at IronWolf if I'm not getting the Pro models.
@hkalisvaart
@hkalisvaart 4 жыл бұрын
Great video, thank you all for all the hard work.
@tommihommi1
@tommihommi1 4 жыл бұрын
holy crap, these drives would even be shitty for consumers. SMR is the QLC of hard drives
@RN1441
@RN1441 4 жыл бұрын
QLC with a DRAMLESS controller.
@albundy7718
@albundy7718 4 жыл бұрын
This is not far from the Truth because SMR, as i understand it, uses a Feature similar to TRIM on these Drives.
@tommihommi1
@tommihommi1 4 жыл бұрын
@@RN1441 well, these drives actually have DRAM cache to mitigate the shittyness of SMR somewhat, or am I misunderstanding something?
@TechyBen
@TechyBen 4 жыл бұрын
@@RN1441 I've got a Samsung drive which is AFAIK TLC or QLC. No problems with it. But it was advertised as their SSD with that tech, they were not bait and switching.
@cataria3903
@cataria3903 4 жыл бұрын
@@TechyBen TLC is MUCH faster than QLC, lots of TLC drives are able to hold over 60% of maximum write speed for the whole drive write. some fall off strong, but no comparison to QLC garbage. TLC and QLC are 2 different classes of drives, of course assuming decent controllers and dram cache.
@davidg4512
@davidg4512 4 жыл бұрын
No more WD red. Go Ironwolf for now.
@dan8t669
@dan8t669 4 жыл бұрын
Good luck with that. Seagate was the first vendor advertising SMR for raid arrays. It's amazing how everyone seems to have forgotten what happened just a few years ago. I remember two companies in 2015 were throwing out all their Seagate SMR drives because they kept dying after ~3 months. Big stories back then, but I didn't save a bookmark... google: *failure rates Seagate 2017* for fun times
@davidg4512
@davidg4512 4 жыл бұрын
@@dan8t669 At least seatgate stayed true to the ironworld series since it's a nas drive. WD on the other hand is like screw that, I don't care if they are nas drives, we are going to put SMR on them and get more money.
@Mythricia1988
@Mythricia1988 4 жыл бұрын
@@dan8t669 IronWolf drives are not SMR, and are very good drives. I have a bunch that have been spinning for 5 years straight.
@manekdubash5022
@manekdubash5022 4 жыл бұрын
It's not quite that simple: above 6TB, they're not SMR any more. You could buy bigger drives instead.
@davidg4512
@davidg4512 4 жыл бұрын
@@manekdubash5022 I have WD 8 TB and they are good
@tangofan4u
@tangofan4u 4 жыл бұрын
Even if you pay attention and buy the CMR version of a WD Red 2-6TB drive, in case of a warranty replacement you'll very likely get an SMR drive. So perhaps best to avoid the WD Red line altogether? Not too many alternatives though...
@scoty_does
@scoty_does 2 жыл бұрын
Seagate and Hitachi
@billgreenwood
@billgreenwood Жыл бұрын
Agree, I sent my SMR drive back and had another SMR back as a replacement. This was a few years back when this issue came to light; it seems that (at that time) WD staff did not know about this issue. Moved to Ironwolf now.
@ShaferHart
@ShaferHart 10 ай бұрын
Right now they're being straightforward about which are smr and which are cmr. CMR priced higher so as of now they can't do this since they're a different series now. I'm guessing this happened after the scandal.
@hardwarexpert
@hardwarexpert 4 жыл бұрын
I only learned of the drawbacks of SMR drives a couple of months ago. I've been extremely lucky in the fact that every drive I've purchased over the years have been CMR drives - including a pair of 12TB WD120EMFZ drives I recently purchased. I run openmediavault for my NAS/file storage needs, I'm also old school in the sense that I only run RAID1 arrays on a P410 SAS controller but its never let me down thus far so I have no need to change.
@hikingpete
@hikingpete 4 жыл бұрын
Thanks for all the hard work. I actually have a few 4TB WD Reds on my desk right now. Thankfully they seem to be of the CMR variety.
@B4dD0GGy
@B4dD0GGy 4 жыл бұрын
27mins of great information, thanks STH
@esra_erimez
@esra_erimez 4 жыл бұрын
Thank you for this. Awesome content as always.
@semirauthsala6001
@semirauthsala6001 4 жыл бұрын
Excellent test and video, Thanks STH
@tekguy5006
@tekguy5006 4 жыл бұрын
Outstanding information in this video. Thank you very much for posting this.
@dizydiz
@dizydiz 4 жыл бұрын
Red drives screwed me and WD support could have cared less. Cost me a couple grand to see if Data could have beee restored at a Tier 4 facility. They couldn't. WD lost a long time customer who has spent $10's of Thousands of $ with them.
@Alphahydro
@Alphahydro 4 жыл бұрын
This is good info Pat. I’m approaching the point where I’ll need to update my ZFS storage from 2 to 4 TB drives and was in the dark about the affects of mixing Red models. Considering stores aren’t open to verify the model being purchased, I’m considering jumping straight to 8 TB models.
@LeonardTavast
@LeonardTavast 4 жыл бұрын
I don't see a use case for SMR outside of the enterprise market (where it can fill a similar roll as tape). It should most definitely not be marketed towards consumers but WD even have a model of 2.5" Black series using SMR. The whole hard drive industry should be ashamed for their false marketing and they should make introduce new product lines for SMR where it is obvious for any buyer that these drives have problems with modifying data. Consumers pay with performance instead of with cash when they buy SMR drives.
@TechyBen
@TechyBen 4 жыл бұрын
Backup drives are fine. Some system drives, as consumers might not be writing more than 50gb random/sequential in one go (a 200gb game or lots of Netflix might be sequential enough or not suffer much from a slow down). They have variable performance, and for someone buying it for NAS they *need* to know if it's a consistent or variable performance. WD hid the fact.
@MyName-tb9oz
@MyName-tb9oz 4 жыл бұрын
I have been so utterly underwhelmed by WD's decisions for the last dozen or so years. I have no trust in their products at all any more. They used to be the gold standard. What a shame.
@Grrizz84
@Grrizz84 4 жыл бұрын
That sort of resilver time alone makes it not a NAS drive 😬
@hamiltonharper
@hamiltonharper 4 жыл бұрын
Man, a few days ago I had to buy 3x 6 Tb Ironwolfs and I was seeing a lot of chatter about SMR. I didn't know much about it other than it was bad, so I just quickly verified the Seagate line didn't use that tech and made my purchase. I thought it was a minor thing but it looks like I dodged the bullet here. I've always avoided WD anyway but damn, I'm glad this information is getting out there.
@vitoswat
@vitoswat 4 жыл бұрын
Finally good test, with conditions that mimic real usage.
@cutterboard4144
@cutterboard4144 3 жыл бұрын
Thank you. And thanks to the Hard Drive Manufacturers that now clearly label their drives as SMR or CMR. Im fairly new to private Disk Arrays, building my own soon, and such reports help learn the important facts regarding this matter.
@Zulatek666
@Zulatek666 4 жыл бұрын
Crap, I so just did this. I had a WD Red NAS array and replaced one drive without realizing they were SMR vice CMR. I found this video a few days after purchase and installing. I am going to go and drop another 105~ dollars and buy a CMR drive. Great video!, sometimes you learn the hard way.
@mhavock
@mhavock 2 жыл бұрын
good video! Also, it might be a good idea for ZFS or such file systems to CHECK for SMR drives before RAIDing them and then warn the user. eg It can have a list of the models etc. The NAS software could also warn you based on the list. I also did not realize this until after I purchased some RED drives...
@ShaferHart
@ShaferHart 10 ай бұрын
This is some great tech journalism kudos to Will and STH. I was looking at two 6tb red drives on Amazon and was wondering why they had a $40 difference. It was CMR. Red drives are very quiet I already own 2 3tb REDs CMR and they've been fine for years (AFTER rmaing a few in a short timespan in the beginning lol) but I do know they're quiet which I took for granted so I'm buying a few more now.
@ServeTheHomeVideo
@ServeTheHomeVideo 10 ай бұрын
Agreed.
@amirjubran1845
@amirjubran1845 4 жыл бұрын
The worst part of this is that only the smaller sizes use SMR, you know, the types of drives that home users with small arrays would use (i.e. arrays with the lowest redundancy).
@TechyBen
@TechyBen 4 жыл бұрын
Consumers with the least technical knowledge to notice they got scammed. :(
@Morpheus-pt3wq
@Morpheus-pt3wq 4 жыл бұрын
Redundancy for home use? It´s obsolete. I´d rather have 8TB for data, than 1x 4TB mirrored. With external backups of course.
@ernestoditerribile
@ernestoditerribile Жыл бұрын
@@Morpheus-pt3wqin my NetApp array I have 9.8PB of mirrored storage. I hate cheap ass TrueNAS,FreeNAS, or Synology solutions, because NetApp is way better.
@ShaferHart
@ShaferHart 10 ай бұрын
@@ernestoditerribile jesus h christ. How much are you paying for that?
@manuelthallinger7297
@manuelthallinger7297 4 жыл бұрын
Sad to hear, especialy WD destroyed such a good branding with this shit. I myself used WD - Red and WD Blues but in my next NAS, there wont be a WD in it, out of principe
@ShOookYx
@ShOookYx 3 жыл бұрын
What other good alternative
@jolness1
@jolness1 4 жыл бұрын
Thanks for doing this testing. So happy to have you guys as a resource as I navigate my first home server build. Thankfully I would have avoided this as I am planning to use 10tb Reds but now I am wary of WD and my be shelling out big money for some HGST drives (even though they are WD) Would you guys still recommend shucking some WD externals (as long as they are larger) or should I start looking for segate to go on sale?
@ernestoditerribile
@ernestoditerribile Жыл бұрын
Go for Toshiba S series instead if you really want reliable drives
@STriderFIN77
@STriderFIN77 3 жыл бұрын
Thank you for this video, as i read this from your website also, and this cleared a bit, heh.. and the thing with 2 SMR 'nas, drives inserted to raid.. i wonder if double the 9days o.O
@johnyu88
@johnyu88 4 жыл бұрын
This is pretty damning.
@Rogue136
@Rogue136 4 жыл бұрын
So unfortunately I do have SMR drives in my system already. I have been no longer buying these drives for the last several expansions as I upped my capacity as the price points changed. Say I have a SMR RAIDz2 vdev and have a drive die. If I replace the dead drive and the new drive is CMR drive will I see these long rebuild times? Basically slowly moving my current SMR drives to CMR as they die. I think this would be the case as SMR should be fine for read performance but I'd love if STH could test this for myself and others who are in this situation.
@kombinezon
@kombinezon 4 жыл бұрын
Thanks for shining more light at that issue. It should be spread as a educational piece for all consumers and why you shouldn't be one. Was adding one 4tb drive into my home nas few months back. It was already populated with Reds so another Red was a no-brainer. I've been involved with IT ever since I was 8. Microsoft, Nvidia, AMD, Verbatim, Sony and many others that I don't remember from the top of my head pulled that sheet and got caught red handed. I have enough experience within IT and freaking shopping in general, that change of one letter in the product designetion lit up red flags everywhere. WD marketing boss must still be feeling the 80s with that kind of bravado.
@lukassvardkvist9161
@lukassvardkvist9161 4 жыл бұрын
0 dislikes a day after upload! WD representative hasn't watch this video yet good work! spread the word!
@tyta1
@tyta1 Жыл бұрын
FYI: Recently bought a WD MyCloud EX2 Ultra and it shipped with two WD Red SMR drives, made mid 2022... so obviously they still think SMR drives are just fine for NAS usage, and/or they're attempting to get rid of their stock of SMR Red drives by putting them into their own NAS models and hoping that customers won't notice or care.
@XDSDDLord
@XDSDDLord 4 жыл бұрын
I was telling my friends this as soon as it came out. We all run FreeNAS. I needed to replace a drive and if the news didn't break when it did, I would have had one of those SMR WD Reds. If I experienced such long rebuild times I would have ended up RMAing it with cross shipping, and there would be a lot of issues. In the end, I ended up buying a WD Red 8TB, since eventually, I want a 6x6TB RAIDZ2 vdev in my pool, but seeing at how 6TB is SMR now, 6x8TB is my only option. I didn't want to spend so much money on these drives, I can't afford it, so it means it's going to take longer for me to finish this upgrade, but unfortunately for me everything I have is WD, and I don't really trust Seagate (the numbers, in my opinion, haven't added up). I'm glad the tech community raised the alarm as soon as it happened to I knew to look into this.
@fractalsauce
@fractalsauce 4 жыл бұрын
At least Newegg is now specifically stating which models of the WD Red series are SMR and which are CMR. Props to them
@vaguedirector_7342
@vaguedirector_7342 4 жыл бұрын
Interesting that these results differ so much from the ones obtained by NASCompares. In his results the SMR drives took ~2x longer when full, and were the same when empty. Does ZFS do something different to regular RAID when rebuilding?
@lloydmilton
@lloydmilton 4 жыл бұрын
so how do the SMR drives go with NON ZFS raid arrays???? to me that is an important question... Reason I ask is I am currently using SAS drives on my server, and SAS drives in my storage enclosure but later on (when the sas drives eventually fail) I will be looking for replacing the sas units with sata units.... My current setup does not use the ZFS file system.... Is this only an issue for ZFS?
@narkoid
@narkoid 4 жыл бұрын
I'm running a supermicro dual Xeon e5 4u 24bay centos 7 rig with an lsi megaraid card with 22 14tb ironwolfs raid 60 3 strips of 7 drives each for the nas storage with 1 hot spare. Then 2x 1tb raid 1 for 2 vm's and the os runs off a 128gb sata SSD. My initial build of the array took 3.5 days. A few months back I had an ironwolf drive that was getting a little loud and making more vibration than I liked. When finally got the guts to swap it out it took 31 hours to rebuild over the weekend with minimal usage except a scheduled off-site incremental backup. I couldn't imagine the stress of that pushing past the 2 day mark.
@tyrannicalmonk1169
@tyrannicalmonk1169 2 жыл бұрын
I know this is an old video but would using a WD Red SMR drive be ok to back up a NAS while rebuilding the array with the older CMR drives?
@ServeTheHomeVideo
@ServeTheHomeVideo 2 жыл бұрын
Probably OK for that purpose if used as a single drive
@monchiabbad
@monchiabbad 4 жыл бұрын
Did you factor in the smr-group-cylinder-cluster size?
@johnsmith9205
@johnsmith9205 2 жыл бұрын
What if the ZFS array is built with SMR drives initially, and then we use CMR drives to replace failed ones? What would be the rebuild time then? (Say, a 3+1 configuration with 4 TB drives). That way we could save money buying the initial drives upfront, and pay more for CMR replacements later as the drives start to fail.
@seedz5132
@seedz5132 4 жыл бұрын
So, that means I had a good "nose" a year ago buying WD REDs for my professionnal back up boxes Is there a good source to know what drives are SMR ? thinking about Seagate here. I know ex Archive drives are SMR, but what about the rest of the EXO family ?
@dany_2945
@dany_2945 3 жыл бұрын
is SMR okay for older games, gigabit Ethernet, music, photos, and things like that? i want to get a 6tb SMR hdd, but I'd rather get a 4tb CMR one if the performance degradation is noticeable
@WatsitTooyah
@WatsitTooyah 4 жыл бұрын
WD shows this on their site now. Apparently there are SMR and CMR models for the regular red non-pro.
@aamiddel8646
@aamiddel8646 4 жыл бұрын
Hi I have a WD red 4GB. Just a single drive in a 2-drive system. Sometimes i notice that the NAS is often accessing the drive hard, minutes at a time while i am not using the NAS (reading/writing to NAS). In my earlier system with a WD red 2 GB i never had this behavior. Has this also to do with this SMR/CMR issue? And since when is WD shipping these SMR drives? Hope you can answer this.
@ServeTheHomeVideo
@ServeTheHomeVideo 4 жыл бұрын
There is a good chance that what is happening is that in the background data is being moved from the "cache" area of the drive onto the SMR portion. That is a common observation those using these drives have.
@jewelbriard5444
@jewelbriard5444 4 жыл бұрын
How about on a basic 2 drive nas raid 1 setup who already this past december 2019 bought these 4TB smr reds for a new nas? I am thinking it is ok to be using them on a new nas, but when replacing a drive , replace with something else. Like many, I also fell into the trap of relying on past history.
@timrightnour4022
@timrightnour4022 4 жыл бұрын
So glad I switched all my new buys to Seagate. Wd has always been a little shady but this is insane.
@namyun2743
@namyun2743 4 жыл бұрын
Be careful with Seagate too. My last 2 Barracuda Pro 2.5" drives were SMR.
@PacAnimal
@PacAnimal 4 жыл бұрын
I used to run WD red 4x6TB in my ZFS raid. Every scrub, ZFS would repair between 64 and 256 KB, always in different places. Scary. I've had no more issues after switching to Seagate IronWolf Pro drives.
@sonny_z900
@sonny_z900 3 жыл бұрын
Wow, seeing this now as I'm orientating in getting a new NAS with WD RED's in it and looking at the 4TB 256mb version of the disk. The 64mb version is CMR. Would you recommend these over the SMR or skip WD in general? I've have bad experiences with Seagate in the past and rather stick to WD instead. What about Toshiba's N300 disk? Thanks.
@ServeTheHomeVideo
@ServeTheHomeVideo 3 жыл бұрын
WD CMR is OK.
@Stoney_Eagle
@Stoney_Eagle 4 жыл бұрын
So to answer my last question. I don't have to worry about my currently running drives to fail but I need to make sure my replacement drive is a CMR drive?
@jensdroessler3575
@jensdroessler3575 4 жыл бұрын
Some time ago I got my hands on an Adaptec 72405 for almost nothing. So I said „let‘s RAID5 what‘s around“, which were some Seagate Archive 8TBs and some new Toshiba N300 8TBs, eight drives in total. It worked so far, RAID build times were like 12 hours or so. I didn‘t think about the implications of mixing CMR and SMR at that time, but seeing your video now makes me think I should swap the Seagate SMRs for more N300s or similar. OTOH every swap will necessitate a rebuild... Oh boy oO
@dashtesla
@dashtesla 4 жыл бұрын
I'm also wondering about the SAS drives market, I personally either go with shitty Seagates 5TB from a backup plus portable or full blown SAS HGST or EXOS from Seagate, haven't used WD much as of lately myself, but i'm curious to know what else is also SMR. I really don't like the sata drives for anything serious, always preferred sas myself.
@schmitzi99
@schmitzi99 3 жыл бұрын
Can someone test how SMR drives fair with sequential resilvering? I would like to know if it makes a difference. Thank you
@alexandrecouture2462
@alexandrecouture2462 4 жыл бұрын
I had to replace the 2 2tb hard drives on my home-made nas using ubuntu and mdadm. I had so many trouble with the newer smr drives that I thrown them and bought HP enterprise-grade hdds. mdadm will always end up marking as bad smr drives, either Seagate or WD.
@abcdefg9613
@abcdefg9613 4 жыл бұрын
I don't really follow the high enterprise market for hdd but I am interested in the purple line since I install on a regular basis surveillance systems. Is the purple line affected? It would be counter productive since these drives do continuous writes . Thanks.
@AndrewFremantle
@AndrewFremantle 4 жыл бұрын
See the WD blog, they now document which of their drives are SMR. If memory serves the Purples _were_ SMR drives. Unlike ZFS, surveillance systems are probably mostly doing sequential writes, and not fully saturating the disk, so SMR can probably tolerate that workload.
@CrkdLtrN
@CrkdLtrN 2 жыл бұрын
I did not know about this. I was curious why rebuilds had been taking a long time though. I logged into my Synology device and went to Storage Manager and lo-and-behold, I have EFAX 6TB drives in every bay..
@Mr_Meowingtons
@Mr_Meowingtons 4 жыл бұрын
I have 4 3TB WD RED hard drives 5 months ago i got 4 more 3TB WD Reds from 2019... i put them in and they all faild doing a raid expanshuon on raid 6... Lucky i had a back up i built the RAID and then spent 8hr coppying over all the data. After that the raid comtroler was showing drive errors and warning.. I had to return all 4 drives. All my older 3TB drives still work fine....
@TheJoaolyraaraujo
@TheJoaolyraaraujo 4 жыл бұрын
Thank you for your work
@CMDRSweeper
@CMDRSweeper 4 жыл бұрын
Thank you again! It does seem I need to find a new vendor when my Red's kick the bucket. A real shame as they have been reliable in ZFS up until this point...
@mikedoth
@mikedoth 3 жыл бұрын
Is there a website that has a table of all the drives?
@daniel070487
@daniel070487 2 жыл бұрын
I just bought two Seagate Barracuda 2tb. Freshly formatted started Crystal Disk Mark and gotten 220mb/s read 200mb/s write. I just put some games of my steam library on it ~160gb and did the test again only to find out it's now at 10mb/s read and ~100mb/s write. And that's all the time no matter if I let it sit for hours (Taskmanager performance 0%). How can I use TRIM on Windows 10?
@virt1one
@virt1one 4 жыл бұрын
While I don't think a Z2 is going to be very common for a 4 bay NAS, I would expect it to be MUCH more common for an 8 bay NAS. I'd be interested in knowing just how bad a double failure would be in a fully populated 8 bay (4tb ea) array. It'll take a long time to test probably, but still would be interesting to know. THANK YOU so much for looking into this and getting concrete results into our hands!
@EduardoPerezEsteban
@EduardoPerezEsteban 4 жыл бұрын
Problem is not NAS, problem is RAID. These drives rely on knowing which areas are "empty" (and can be overwritten) and which areas are "occupied" (and must be relocated). When the RAID controller initializes the array, it writes all over the drives, so they appear as "full"; then, any subsequent write operation implies relocating some data. The resilver procedure did not take 15 times longer just because the drives are 15 times slower, it tool 15 times longer because all that data relocation required 15 times more work. Not only these drives are much slower in a RAID array, they are going to die pretty quickly.
@s1mph0ny
@s1mph0ny 4 жыл бұрын
That's not how formatting works in any filesystem made in the last 20 years. You're right about quick death though.
@EduardoPerezEsteban
@EduardoPerezEsteban 4 жыл бұрын
@@s1mph0ny Notice how I did not mention anything about filesystem formatting but array initialization in RAID.
@AJL713
@AJL713 4 жыл бұрын
Could you guys make a video explaining ZFS
@brianh.000
@brianh.000 4 жыл бұрын
Just about to click "order" on a WD EFAX when I came across this video. *whew*. So, I see EFRX (CMR) are still available. This is what I currently have. I wonder if I should get one of those, or get a different brand CMR drive... Hmmm...
@djordje1999
@djordje1999 4 жыл бұрын
What about WD MyPassport wdbs4b0020bbk-wesn? I can't find witch technology is used..
@stonent
@stonent 4 жыл бұрын
When I shared this on Facebook it says SMR vs SMR and not CMR.
@tromick
@tromick 3 жыл бұрын
I saw this video like 5 times in one year. I can't stop myself... I will ask it.. I have 10TB music archive. I am sharing my files with Soulseek. My computer is 24/7 open and not always but most of time it does read due to uploads. And sometimes i download new music files. Now... Due to disk plates will be more on higher capacities and i want less head, surface fails, i will buy 4TB drives. In addition i am trying to buy CMR/PMR drives. WD40PURZ/WD40PURX (I don't know which one is PMR/CMR), WD40EZRZ and WD40EFRX are my options. So which one do you recommend me in these? Thank you.
@luanmaia
@luanmaia 3 жыл бұрын
I Just boubas 2x4tb from them.. should i be worried ?
@Scm391
@Scm391 3 жыл бұрын
*Newbie question* Would SMR have the same problems in a ZFS Mirror rebuilds(resync)??
@ServeTheHomeVideo
@ServeTheHomeVideo 3 жыл бұрын
Yes
@MrKillswitch88
@MrKillswitch88 4 жыл бұрын
I wonder how SMR affects the platters in the long haul once platter degradation becomes a factor especially for laptops where capacities have been stagnant for over half a decade. I can live with sata 1 speeds but I deeply loathe data loss and performance degradation as the drives age.
@xfxxgj7086
@xfxxgj7086 3 жыл бұрын
Smr become extremely slow but reliability depends upon batch and manufacturing dates and many things still wd is less error prone but newer segate from 2018 2019 2020 and up are good 2020 v are less problematic still Do this backup the backup yes backip the backup
@kaspersergej
@kaspersergej 4 жыл бұрын
This video is long, yes. But exact description of the test procedure is crucial. However, you repeat yourself a lot.
@AidenPryde3025
@AidenPryde3025 4 жыл бұрын
Is there a compiled list of these SMR drives so that I can avoid them?
@happygimp0
@happygimp0 4 жыл бұрын
Avoid WD
@Partykardi
@Partykardi 4 жыл бұрын
Do We know, if it would have affected the rebuild time for the worse, had all drives been SMR, not just the one that was replaced?
@ServeTheHomeVideo
@ServeTheHomeVideo 4 жыл бұрын
We have not gotten to test this yet, however, the general rule is that an array runs at the speed of it's slowest drive.
@pattoman5568
@pattoman5568 4 жыл бұрын
I don't understand, if you have a drive failed shouldn't you backup your data first before rebuilding?
@billgreenwood
@billgreenwood Жыл бұрын
I had this when it first came to light. I had to do the research. I accidentally bought a 6tb SMR and it was hell. Took an age to rebuild, then performance on my FreeNAS when this resilvered was awful. I'm moving to Ironwolf ST6000VN001 when I need to replace a drive.
@AllanSavolainen
@AllanSavolainen 4 жыл бұрын
Hmm, these results seem to differ from other similar rebuild tests I've seen on YT. In my own tests it seems that having separate SSD CACHE and LOG devices helps alot and is recommened always when using ZFS. And rebuilds on normal RAID1/5/6 setups work just fine on SMR drives.
@berndeckenfels
@berndeckenfels 4 жыл бұрын
resilvering should be more performant if the raid layout fits the shingle sizes, so they are not rewritten?
@AndrewFremantle
@AndrewFremantle 4 жыл бұрын
DM-SMR drives do not disclose their shingle size. Even if you got the shingle size correct, those shingles are _massive_, relative to the 512B or 4kB sectors that zfs was designed for.
@davethenerd42
@davethenerd42 2 жыл бұрын
What about the other way around? Nas made of SMR with a new dmr?
@awesomearizona-dino
@awesomearizona-dino 4 жыл бұрын
I bought one of these Just a week before the story broke. (4Tb) not so happy. Thanks for testing.
@ServeTheHomeVideo
@ServeTheHomeVideo 4 жыл бұрын
Sorry for not posting earlier. We had the results by mid May but wanted to try another drive before posting something like this. Hopefully folks understand why we wanted to double check findings and methodology before posting.
@awesomearizona-dino
@awesomearizona-dino 4 жыл бұрын
@@ServeTheHomeVideo Hi, Thanks for the update... i totally understand..Please let me share 1 more thing. This new 4tb Red. causes extended boot times upto 90 seconds (its not the boot drive) I have an ssd that typically boots in 20 seconds into W7pro 64bit. it happens no matter what sata port it is plugged into. You may want to see if that is another issue these Reds can cause.
@HotDamnHarry
@HotDamnHarry 4 жыл бұрын
@@ServeTheHomeVideo I have FreeNAS raidZ2 6x6tb, (4xWDRed, 2xIronWolf) where one of the Reds has died.. to be replaced with another Ironwolf.... Do you want some rebuild times for this?? and should resilvering onto a Ironwolf pose the same 13x performance hit?? I am planning on swapping out the Red's one at a time eventually replacing them all...
@jorisdevaan6845
@jorisdevaan6845 4 жыл бұрын
love the subject and like the video but everytime I watch this channel I fall asleep :)
@misium
@misium 4 жыл бұрын
1. Why does re-silvering cause so many randomish writes? I thought it would be technically possible to rebuild the drive in a linear fashion. 2. Is rebuilding mirrors also slowed down?
@TechyBen
@TechyBen 4 жыл бұрын
Different systems do different types of resilvering. Some might get file changes as it is resilvered (the server still in use), and so a SMR drive will seriously have difficulty keeping performance when having to do sustained full speed write, and then read/modify/write in the middle of that workload. SMR can do sustained consecutive unmodified reads and writes faster than CMR, but as soon as you try to modify a file during a write load *and* while their cache is full, you might drop as low as 10% the normal performance. Ouch!
@misium
@misium 4 жыл бұрын
@@TechyBen So does it mean that smr drives are "safe", if you take the array offline for re-silvering? This is a scenario that might be good enough for a home server.
@TechyBen
@TechyBen 4 жыл бұрын
@@misium No idea. If the companies are not being honest with the customers, why should we put in all the hard work to make sure it's easy to use? I'll go shop elsewhere and decide when and how I get CMR or SMR drives. :)
@philyoung4238
@philyoung4238 4 жыл бұрын
Has anyone done any testing with the 10TB white label WD drives? SMR or CMR?
@ServeTheHomeVideo
@ServeTheHomeVideo 4 жыл бұрын
Phil, this is something I want to see as well. The challenge is getting the time/ money to do a project like that. It is on our list of potential topics to explore, but it is a lot to invest.
@rhangman8284
@rhangman8284 4 жыл бұрын
Recently swapped a bunch of 3TB drives (mix of WD Red and Seagate NAS) for 6TB Reds Speed went from "scan: resilvered 2.55T in 1 days 03:44:02 with 0 errors" to "scan: resilvered 2.56T in 1 days 06:11:03 with 0 errors". Were all WD60EFAX-68SHWN0. 11 drives replaced fine. Didn't consider a bit over a day as too bad. 12th drive failed with write errors though. Ran badblocks, smart test, WD Diag, all good, so tried again, few errors and then got stupid slow, so detached. Tried again with a replacement drive, this time a WD60EFAX-68JH4N0. Once again, drive all good with badblocks, etc. was going fine at a decent speed till faulted with write errors again. Thinking that the 12th WD60EFAX-68SHWN0 was just faulty and the WD60EFAX-68JH4N0 is technically fine, just with the SMR being so slow, writes timed out causing ZFS to label it faulty? Believe all the WD60EFAX-68SHWN0 should be SMR though? Definitely got slower the more of them that were in the array, but not 15+ times slower. In my case, was replacing active drives though, not faulted/removed ones. So not sure if the not so bad speed was that or not all EFAX drives as bad as each other? Might be worth testing replacing active drives as a comparison? Interestingly, same AsRock Rack board, although PCI-E HBA, replacement drives using MB SATA though before being swapped.
@katrinabryce
@katrinabryce 4 жыл бұрын
I just won't be specifying WD Reds in any future builds. Even if I pick the right model number, can we trust someone in a warehouse to pick the correct product from the shelf? The other use case is that someone is buying a new NAS, and then they will migrate their data from their existing smaller capacity NAS, or they want to organise their existing data from desktop storage and external USB devices into a central location. Most businesses don't start with a NAS from day one. It is something they do when they reach a certain size and their data starts to get unmanageable. So it would seem to me that any NAS drive is going to have a period of high-volume workload when it first comes out of the box. Data migration is something you need to be able to start running on Friday evening and come back to on Monday morning to do the final steps before starting work for the week.
@murpmich59
@murpmich59 4 жыл бұрын
What about 3tb? I bought 7 wd reds 3tb and ran in freenas the last few years. Lost 1 after a year and just recently found out the 3 more are bad in my server. Shut server down. Tested and they have bad sectors. Why? I hardly used the server, the drives just sat there idling for a couple years.
@chrismoore9997
@chrismoore9997 4 жыл бұрын
I found this in testing over a year ago. Been thoroughly discussed on the FreeNAS forum. You should check it out.
Why are Drive Manufacturers in BIG TROUBLE?
12:22
Linus Tech Tips
Рет қаралды 2 МЛН
ОСКАР vs БАДАБУМЧИК БОЙ!  УВЕЗЛИ на СКОРОЙ!
13:45
Бадабумчик
Рет қаралды 4,6 МЛН
Wait for the last one! 👀
00:28
Josh Horton
Рет қаралды 160 МЛН
Why We Need SMR Listed in Hard Drive Specs
23:50
ServeTheHome
Рет қаралды 41 М.
Choosing The BEST Drive Layout For Your NAS
21:42
Hardware Haven
Рет қаралды 118 М.
Comparing 9 Popular SFP+ to 10Gbase-T Adapters
11:03
ServeTheHome
Рет қаралды 104 М.
QNAP Put TWO ZFS NAS Units in 3U
25:01
ServeTheHome
Рет қаралды 24 М.
9 Steps to Recover from a Proxmox VE Boot Disk Failure
32:51
ServeTheHome
Рет қаралды 24 М.
WD Red NAS Hard Drives - Before You Buy
22:23
NASCompares
Рет қаралды 43 М.
THIS is the Future of Storage in Servers
17:58
ServeTheHome
Рет қаралды 151 М.
Are SMR Hard Drives still 'BAD' in 2024?
11:46
NASCompares
Рет қаралды 10 М.
После ввода кода - протирайте панель
0:18
Up Your Brains
Рет қаралды 1,2 МЛН
YOTAPHONE 2 - СПУСТЯ 10 ЛЕТ
15:13
ЗЕ МАККЕРС
Рет қаралды 186 М.
WATERPROOF RATED IP-69🌧️#oppo #oppof27pro#oppoindia
0:10
Fivestar Mobile
Рет қаралды 17 МЛН