Tour of DrivePool M4

Alex · December 2011

I've just posted a complete tour of DrivePool M4 UI as it is today (dev build).

http://blog.covecube.com/2011/12/a-tour-of-stablebit-drivepool-beta-m4/

I guess it's customary to announce things is short form these days, so for those of you who don't like to read too much.

What's new:

An awesome new file system, CoveFs, specifically designed for pooling. 100% native, on par with NTFS. No reliance on any user mode service. Needless to say, this is very big.
DrivePool is now a disk. Yes, it has a drive letter.
Deeper integration with the Dashboard.
Usability improvements such as background duplication changing.
Indexing support.
Refreshed UI. Read the blog post for more...

What's taking so long?

I've decided to cut loose the open source 3rd party file system used in M3 and write my own specifically optimized for pooling.

As some of you may know, the original plan was for a performance layer to run on top of M3. While implementing the performance layer it became obvious that making this layer be the file system itself would yield enormous benefits such as stability, compatibility and performance.

It would take a bit longer, but after doing some testing it was the obvious choice. I had to write a new kernel virtual disk driver and a new file system. But rest easy, these have been done for a while now. They might need some tweaks before release, but my main focus right now is to bring all the pieces together. I'm shooting for the end of this month for a public release. If I miss it, then it will be Jan.

Overall, it's looking good now. It was a bit of work to get here so hopefully people will find it useful.

Sorry if I was neglecting the forum, but there are so many hours in a day. You can always get in touch with me via. stablebit.com.

ajuhawk · December 2011

Thanks for the update, much appreciated! Looking forward to testing the fruit of your labors.

olgemap · December 2011

I read through your overview and it sounds like great improvements have been made. I was a little worried about a new file system, but you specifically stated that it's fully backwards compatible with NTFS, which makes sense.

What about the conversion from M3 to M4. I had, just today, decided to bit the bullet and install M3 and worry about M4 later since I've assembled all the hardware I need. Since there now is a timeline for the M4 release, I'm back decision table again. What does it take to move from M3 to M4? Is it a fairly easy process in the sense of just uninstalling M3 and installing M4 (which I assume would convert the files to the new CoveFs)?

Awesome looking software by the way, can't wait until the RC is out.

Danee · December 2011

Looks very good! Can't wait to test it!

Cheers,

Danee

m2and3ingo · December 2011

will M4 upgrade over M3?

M2nad3ingo

snake · December 2011

It sounds like a lot works for you in short amount of time. Good job!

Codegear · December 2011

Thank you Alex for providing feedback. We were all waiting to hear from you!

Klode · December 2011

That is a great overall! I like the deeper integration with the dashboard. I think it will be less confusing for people. Can't wait to try it out with the new UI. Keep up the good work!

Claude

coalfield · December 2011

looks awesome. Love the idea of having an actual drive letter. Keep up the good work!

rubenhak · December 2011

Alex, new UI looks perfect!

Could you also provide some numbers to get a rough idea how will M4/Release version operate? Currently I'm getting 10-12 MB/s write and read performance on a wired Gigabit network. Running on i3-2100T CPU with plenty of memory. My network was tested to easily transmit up to 80MB/s.

Just want to get idea what to expect from final product.

m2and3ingo · December 2011

I Have been awaiting this news a a long time. Thank you for the information and keep up tye good work

Rufus Roper

Klode · December 2011

Currently I'm getting 10-12 MB/s write and read performance on a wired Gigabit network.

I just tested my network speed when copying 6GB of data to a non-pool drive and I'm getting between 45-70 MB/s. If I copy to a pooled drive, I get 24-27 MB/s over a wired Gigabit network. My server is an EasyStore H341 with 4 GB memory. May be you need check your setup and re-test your speed.

Claude

rubenhak · December 2011

Klode, I'm actually using augmented Acer EasyStore H340, which now ASUS P8H61-I motherboard, Intel i3-2100T and 8GB DDR3 1600Mhz RAM under the hood. This thing is a hell lot more powerful than original system with crappy Atom CPU. On a system drive 1TB WD Black Series, + 3x 1TB WD Green HDDs. Network is wired and supported by one of the best Cisco Gigabit switches available.

Could it be because of my green HDDs?

rubenhak · December 2011

Klode, thanks for telling to check the setup. It turned out that I mistakenly set 100MBps limit on my network card on the storage. It was the time when i wanted to set the link speed and duplex to "1GBps Full Duplex" but most probably did a mistake and set 100MBps

Now, with 1Gbps set, I get 90-100MB/s read as well as write access to shares, no matter if those are pooled or non pooled shares. Very happy now!

Johre · December 2011

Klode, I'm actually using augmented Acer EasyStore H340, which now ASUS P8H61-I motherboard, Intel i3-2100T and 8GB DDR3 1600Mhz RAM under the hood. This thing is a hell lot more powerful than original system with crappy Atom CPU. On a system drive 1TB WD Black Series, + 3x 1TB WD Green HDDs. Network is wired and supported by one of the best Cisco Gigabit switches available.

Could it be because of my green HDDs?

I am currently running with 4 Westerdigital (WD20EARS) green drives and i avg around 30MB/S writing over a wired gigabit network. The only thing i can think of is that mine go through an old LSI raid controller and an Intel SAS/SATA expander wich maybe handle data a little better then just through a MB sata controller.

rubenhak · December 2011

Johre, check my post above. My problem is solved. I mistakenly limited NIC card to utilize 100Mbps instead of 1Gbps. Not it is flying 90-100MB/s read and write.

disappointed · December 2011

DrivePool is now a disk. Yes, it has a drive letter.
Deeper integration with the Dashboard.
Usability improvements such as background duplication changing.

Wow after all this time and effort you seemed to have created a DriveBender clone. I dont get it, M3 was almost there, my guess is now we are in for another long round of testing.

hellerbrewing · December 2011

Check out this review of Drive Bender:

http://www.mediasmartserver.net/2011/11/04/review-drive-bender-for-windows-home-server-2011-part-1/

After reading through all of the caveats and warnings, I decided that something that was dealing with my crucial data should not be so sensitive to being set up perfectly. I think everyone here will agree that drivepool, even in M3, is a simple an elegant replacement for drive extender. From what I have seen of M4, it has only gotten more simple and elegant. IMO this is even farther removed from what drive bender is, not closer to it. I have not seen one warning about drivepool saying "make sure you do this or it won't work."

Philmatic · December 2011

DrivePool is now a disk. Yes, it has a drive letter.
Deeper integration with the Dashboard.
Usability improvements such as background duplication changing.
Wow after all this time and effort you seemed to have created a DriveBender clone. I dont get it, M3 was almost there, my guess is now we are in for another long round of testing.

I don't understand how it's a DriveBender "clone"? Having the pool mounted as a drive letter pretty much opens DrivePool to 100% compatibility with WHS. If WHS can see the drive as a drive letter, it's going to assume that it is a normal hard drive that it supports natively. That's nothing but good there.

Deeper integration with the WHS console is another huge benefit. It may be "DriveBender like", but who cares, are you suggesting that he doesn't implement features that his competitors have?

I used DriveBender initially for my new server set up, I did NOT like it. I had major problems setting it up and it never worked quite right for me.

DrivePool has it's problems, but I feel its direction is more in line with what I would like to see in my pooling software so I'm sticking to it. If they can fix:

Permissions issue with shares, it is getting very annoying to have to force the resetting of all my files almost every day.
Clean up the GUI, fix the sorting logic and maybe make it look more like StableBit Scanner (Extremely pretty BTW)
Better handling of failing/failed hard drives, this drove me NUTS
Implement the other features they've already announced

I have NO problem forking over $40 for this great piece of software.

rubenhak · December 2011

Agreed with Philmatic. I would just add:

5. Fixing temporary access denied issues caused when moving lots of files/folders around.

and this should be good to go!

As mentioned in earlier posts, yes I believe it should be fully retested. M4 sounds like to be a brand new software.

disappointed · December 2011

Check out this review of Drive Bender:

http://www.mediasmartserver.net/2011/11/04/review-drive-bender-for-windows-home-server-2011-part-1/

I'm not sure how relevant an outdated review is, this looks nothing like the software I've just installed.

DrivePool has it's problems, but I feel its direction is more in line with what I would like to see in my pooling software so I'm sticking to it. If they can fix:
Permissions issue with shares, it is getting very annoying to have to force the resetting of all my files almost every day.
Clean up the GUI, fix the sorting logic and maybe make it look more like StableBit Scanner (Extremely pretty BTW)
Better handling of failing/failed hard drives, this drove me NUTS
Implement the other features they've already announced

Agreed with Philmatic. I would just add: 5. Fixing temporary access denied issues caused when moving lots of files/folders around.
and this should be good to go!
As mentioned in earlier posts, yes I believe it should be fully retested. M4 sounds like to be a brand new software.

This is my point, M3 was so close a number of bug fixes and additions and it would have been GTG. But now who knows how months of testing are ahead, and the end result will be a clone of an existing product, only DP will be 6 to 12 months behind in terms of product maturity. I installed DriveBender yesterday and it was easy to setup and performs well, other than some pie charts in M4, what's the difference?

Sorry if I sound negative, but if you installed and used DriveBender, you would see my point.

rubenhak · December 2011

disappointed, it was clear from the very beginning that DrivePool and DriveBender would get very similar feature sets. There are some glitches with DrivePool, and also it turned out that some decisions made on the core concepts used in the architecture did not turn out to be best ones and now M4 is a lot different from M3. Personally being a software engineer, this makes me like Drive Pool better. That would only mean that final product would be even more polished.

For now I'd stay with M3, as I'm using it for storing my real data and I do not have a backup of it elsewhere. So 4133 works for me. I think I will wait for the final release and then would install it.

Klode · December 2011

Klode, thanks for telling to check the setup. It turned out that I mistakenly set 100MBps limit on my network card on the storage. It was the time when i wanted to set the link speed and duplex to "1GBps Full Duplex" but most probably did a mistake and set 100MBps
Now, with 1Gbps set, I get 90-100MB/s read as well as write access to shares, no matter if those are pooled or non pooled shares. Very happy now!

Hi, knew there was something wrong. No matter what powerful server you have, it shouldn't matter on the network speed. I know it wasn't the right thread to discuss, but couldn't let it go before adding my own experience and try to help out the user. Happy you got it working!

Claude

Shane · December 2011

disappointed said:

"This is my point, M3 was so close a number of bug fixes and additions and it would have been GTG. But now who knows how months of testing are ahead, and the end result will be a clone of an existing product, only DP will be 6 to 12 months behind in terms of product maturity."

Sometimes you have to backtrack before you can successfully continue the ascent. But that doesn't mean you've also tossed away everything you learned along the way - the "path you shouldn't have taken" can often teach you a great deal about what the proper path actually is.

I note that DriveBender still has a few pending fixes and additions of its own. And I'm not surprised that they have similar UI - they're both blazing trails up the same mountain. What remains to be seen is just how rewarding each trail will be for those who take them.

Shane · December 2011

Noteworthy tidbits from Alex in the blog comments:

(re adding server folders into the pool) "M4 was deeply re-designed to address folder moving issues."
(re moving from M3 to M4) "There will be an upgrade path."

Saltydh · December 2011

I've been using DrivePool with FlexRAID for software level drive parity. With the new Filesystem being introduced in M4, will individual pool drives still be made available through either a Drive letter or an NTFS mount point?

I see the obvious speed benefits of a dedicated filesystem however for some of us that will come (potentially) at the cost of a redundancy / backup solution depending on how those software packages work.

Ps. I realise that FlexRAID offers a similar component for drive pooling. I've found Stablebit's to be far more reliable

Thoughts?

Saltydh · December 2011

Damn, don't you hate it when you find a reply just after you post

I found the following on the Blog for M4 with regards to the CoveFS:

"...Is the filesystem on the disk still native NTFS?"

Alex's reply:

"Yes. Absolutely.

The files are still stored on the individual disks at standard NTFS files. You can still plug in a pooled disk to another machine to get at your pooled files.

CoveFS is simply the file system that facilitates disk pooling. So when you copy a file to the pool, it goes onto one of your NTFS disks, and the file is completely unaltered. Duplicated files go on 2 disks.

CoveFS doesn't filter I/O or modify I/O that goes to your regular disks. It's not involved there at all. So when you copy a file to a non-pooled disk, CoveFS is not involved in any way."

Awesome news! thanks Alex.

reded23 · December 2011

Check out this review of Drive Bender:

http://www.mediasmartserver.net/2011/11/04/review-drive-bender-for-windows-home-server-2011-part-1/

I'm not sure how relevant an outdated review is, this looks nothing like the software I've just installed.

DrivePool has it's problems, but I feel its direction is more in line with what I would like to see in my pooling software so I'm sticking to it. If they can fix:
Permissions issue with shares, it is getting very annoying to have to force the resetting of all my files almost every day.
Clean up the GUI, fix the sorting logic and maybe make it look more like StableBit Scanner (Extremely pretty BTW)
Better handling of failing/failed hard drives, this drove me NUTS
Implement the other features they've already announced
Agreed with Philmatic. I would just add: 5. Fixing temporary access denied issues caused when moving lots of files/folders around.and this should be good to go!As mentioned in earlier posts, yes I believe it should be fully retested. M4 sounds like to be a brand new software.

This is my point, M3 was so close a number of bug fixes and additions and it would have been GTG. But now who knows how months of testing are ahead, and the end result will be a clone of an existing product, only DP will be 6 to 12 months behind in terms of product maturity. I installed DriveBender yesterday and it was easy to setup and performs well, other than some pie charts in M4, what's the difference?

Sorry if I sound negative, but if you installed and used DriveBender, you would see my point.

Like a lot of people using Drivepool....we did try out Drice Bender to find out it has many more issues than Drivepool ! Not to mension the more complicated user experience.

m1kehunter · December 2011

Drivepool for WHS. Drivebender for other windows versions.

JazJon · January 2012

I think Alex is going down the right path. M3 will never be 100% perfect, and I'm really excited to read about the choices Alex made moving forward into M4. The native full compatibility is my number one request. Right now indexing is not working in M3, and I can't even create Picture play lists etc on my xbox extenders that talk to my media center computer. Once M4 is setup on my WHS, I'm expecting all shares and libraries will finally be fully compatible! I'm also expecting to finally stop getting corrupted backups/database issues, and file duplication problems every few days. I'm really looking forward until things are a true set it and forget it appliance level configuration/installation. and ready for prime time. It's well worth the wait. I too don't even want to try DriveBender, looks too finicky and clunky. I'm sticking with Alex and ready to pay what ever he charges!

Alex · January 2012

As far as the new architecture in M4, I just want to clarify exactly why the changes were made and what the impact is.

First, the way that your files are stored remains the same in M4. In fact, it is identical, unaltered NTFS files. This was very important because you need to be able to plug in the pooled disk to any other machine and be able to read your files. This is a core design principal and will not change.

So what's the new architecture?

Well, it's just the layer that sits between your pool and NTFS. The part that actually pools the disks and presents a unified view of them. In DrivePool this is implemented as a file system, it always has been.

Another way to do this would be to implement a soft-RAID like solution. But there are some consequences to doing that. Namely, your files are stored in a "black box". In other words, it's just a big blob of bits and you can't do anything with them if the system breaks.

So DrivePool has always presented itself as a file system, even though it's utilizing NTFS underneath.

DrivePool M1 to M3 utilized an open source component (Dokan) that is designed to make building new file systems easier. It is a generic layer that sits in the kernel and forwards all requests to a user mode system service.

The plan was to utilize this to help solidify the architecture in DrivePool before "burning in" the code and rewriting it in a custom file system driver.

But for a while there it was working so well, and fast enough that I considered keeping it for 1.0. There was going to be a performance layer (filter driver) to optimize certain I/O paths, like reading and writing, directly in the kernel. This is what we have in M3 now.

But some problems remained.

One problem is that a file system is supposed to be able to start up before any user mode component. This is impossible when you are relying on a user mode service. No other built-in file system relies on any user mode code.

Another problem is because we are "virtualizing" the I/O, subtle compatibility differences arose when compared to true NTFS.

Also, if the user mode service crashes or is stopped, what happens to the pool? Well, it becomes inaccessible.

Plus, I've received a lot of great feedback from people about how useful they find DrivePool. This motivated me to take it up a notch and develop something even better.

In DrivePool M4, Dokan is gone. It has been completely replaced by custom kernel mode driver that is designed to work as closely as possible with NTFS. This new file system is called CoveFS and carries with it none of the problems mentioned above.

The internal architecture of CoveFS is completely different from M3. I.e. there is no user mode service, it's a "real" file system. However what it accomplishes is pretty much the same as what you saw in M3.

At the end of the day, I expect better compatibility / reliability.

Howdy, Stranger!

Categories

Poll

Tour of DrivePool M4

Comments