DrivePool 1.1 and Beyond

Alex

Hi everyone,

Thanks for showing your great support of DrivePool.

I just wanted to let everyone know that DrivePool is moving forward and very cool things are in the works.

Let me give you a rough road map of what's coming:

DrivePool 1.1

I believe that the current BETA merits a minor version bump, so it will be released as 1.1.0.xxxx. This is mainly because CoveFS received a significant overhaul. Other new stuff includes fast drive removal, network I/O boost, and an updated read striping algorithm using blocks.

DrivePool 1.2

I'm looking forward to this build The seeds for it are already getting put in place.

We will have a completely extensible file balancing / file placement plug-in system. It will be extremely simple for anyone with basic programming knowledge to write DrivePool plug-ins that control file placement and balancing.

The plug-ins will be written in .NET.

What does this mean?

For one thing it will let the Scanner do things like automatically migrate your files off damaged disks. It will also give every DrivePool user full control over file balancing. You will be able to set limits on which disks can hold duplicated vs. unduplicated files and how much.

We'll have a few built-in plug-ins, but anyone is free to write any kind of plug-in that they like.

The new plug-in based system will be layered, so you can have multiple plug-ins interacting together to build one balancing model. So you won't be limited to selecting only one balancing algorithm.

What else can be done with this?

A performance optimization plug-in. Let your fastest disks save all the new files, and then you migrate those files to the slower disks at night automatically. You can even use a SSD for this.

Prioritize disk usage by age. Perhaps you don't want to risk putting any unduplicated files on a disk that's more than 3 years old.

Heat management. Place files on the coolest disk, up to a fill limit.

These are just some ideas that sound cool to me. Anyone would be able to write these kinds of plug-ins.

DrivePool 1.3

Multiple pool support.

Yes, you've asked for it and I think we can make it happen with almost no additional UI complexity.

DrivePool 2.0

Major new pooling capabilities that are being worked out now.

----

So that's the road map, and obviously as we move into the future things may change.

SilverRubicon

Fantastic.  With the news of the demise of WHS, have you considered moving the project to Windows 7/8?

Danee

Great news, thanks for informings us and keeping this alive, this will extend the live of whs for a lot of us while microsoft leaves us in the cold!

Alex

Fantastic.  With the news of the demise of WHS, have you considered moving the project to Windows 7/8?

Yes, both DrivePool and Scanner will be ported to run on Vanilla flavors of Windows later in the year.

They will run on Vista an above. DrivePool will probably only support 64-bit OSes.

In terms of the road map, this will happen for DrivePool 2.0.

273c

Any chance of approximate dates for when these versions (1.1, 1.2, 1.3 & 2.0) will be in beta (OK - I know 1.1 is already in beta) and also when released?

Alex

Just roughly,

1.1 should be out any week now (Final). There are still some issues that are being looked at for 1.1.

1.2 is actually already in the works and should follow 1.1 almost immediately (BETA). Most of the back-end infrastructure is done.

1.3 some months later.

2.0 in BETA, not feature complete, by year's end.

----

Scanner development has now shifted over to a non-WHS version, along with a stablebit.com update.

DrivePool for non-WHS will probably happen together with 2.0. But we'll see how it goes with 1.3, perhaps we can make that happen sooner.

Danee

Fantastic.  With the news of the demise of WHS, have you considered moving the project to Windows 7/8?

Yes, both DrivePool and Scanner will be ported to run on Vanilla flavors of Windows later in the year.

They will run on Vista an above. DrivePool will probably only support 64-bit OSes.

In terms of the road map, this will happen for DrivePool 2.0.

Does this include server 2012 essentials?

Philmatic

Server 2012 Essentials would be perfection

hopester

DrivePool 1.2

We will have a completely extensible file balancing / file placement plug-in system. It will be extremely simple for anyone with basic programming knowledge to write DrivePool plug-ins that control file placement and balancing.

It will also give every DrivePool user full control over file balancing. You will be able to set limits on which disks can hold duplicated vs. unduplicated files and how much.

The above will be great when using drivepool with flexraid. You will be able to have duplicated shares on 2 drives for smaller files like docs, photos and music and then have unduplicated shares protected by flexraid on the other drives protect by parity (Static content)

Alex

Server 2012 Essentials will probably not be supported with a custom dashboard add-in.

DrivePool will most likely run on the OS, as a stand-alone application.

Danee

To bad, can you tell us why?

Mathis0

Hello. I'd like to add an idea about a new feature.
Imagine you have a pool of slow but high-capacity low-speed harddisks (like WD20EARS). Those disks throttle the data transfer speed to/from the server. Then you could add some low-capacity high-speed harddisks (like WD Raptor). When transfering data to the server, the plugin would redirect the data onto the high-speed disks, so that the transfer is at maximum available speed. Later when duplication is being started, the plugin would duplicate those new files onto the slower high-capacity disks - and remove them from the high-speed disks (making room for new files). A detail could be like holding often used server-files on high-speed disks, if the user decides, so also reading from the server can reach maximum speed.

Shane

Hi Mathis0, that sounds like it would be covered by the new architecture Alex described, "DrivePool plug-ins that control file placement and balancing."

Drashna

Sad to hear that you aren't planning on supporting Server 2012 Essentials. It contains a very similar dashboard to WHS2011/SBS2011E. In fact, the add-in can be installed, but doesn't work reliably. Neither does the driver.  I imagine that it wouldn't take much to tweak the dashboard. 

The main reason I'm sad to hear this, is that without StableBit, I have no intention to upgrade to 2012 Essentials. Storage Spaces is an extension of the dynamic disk features, and not much more. It's software RAID, with a couple extra features. 

It gets worse too. Creating a "Storage Pool" from the control panel is not compatible with the Server Management console, and vice versa. If you create a storage pool from the control panel, you also cannot remove disks unless they are physically attached. However, you can remove disks if you used the Server Management, but it's fairly confusing on how to.  Also, you cannot shrink the pool, only expand it. Also, some of the advanced features (that make it much more reliable) are only available via power shell commands, IIRC. And there is no way to specify a folder is duplicated or not. It's drive level.

The thing is, there are a number of improvements to 2012 Essentials. Namely, the launchpad isn't a horrible wreck,the alerts aren't obnoxious anymore, and the backup engine is much more efficient on the client side at least.

Shane

Note that Alex only said he probably wouldn't support 2012 Essentials "with a custom dashboard add-in", and furthermore went on to say "DrivePool will most likely run on the OS, as a stand-alone application."

YMMV, but I would be quite satisfied with a stand-alone application (especially if it freed up Alex's time to make DrivePool work on more of the various Windows OS flavours).

ian_spencer

This all sounds quite interesting. My server is costing too much in electricity and I am trying to keep it off, just on for backups every week. I use Crashplan for online backups of changing data so worst case I lose data and do a WHS 2011 restore then download changes from online as a last resort but it is costing me more in internet data and slows the internet.

I installed Windows 8 Pro RTM assuming they had the bugs worked out and their Drive Pool would be well worked out after the debacle of the WHS history, I don't know who they use for focus groups to decide what they are going to do!

Other than drivers that I eventually worked out I went through the largest windows learning jump sideways rather than progression, I have been on every version of Windows from 3.0 to 8 now except 2000 and ME.

I then in the new install process started to find some serious design flaws with Drive Spaces, the biggest being balancing. I can't just move my drives from WHS to my Desktop, it looks like you need to start with all new and same size drives to have any chance and adding extra drives does not seem to be a good option. The more I read the more I worried, Drive Spaces was a key reason for the move. 

Unless you are buying a new computer and have a lot of time on your hands Windows 8 is not ready. It is faster in performance but slower in daily usage. 

I can see what they are trying to achieve but they have themselves a mongrel of a beast now. The apple computer is not like an iPhone / iPad so I have no idea why Microsoft thinks metro on a Desktop makes sense.

I have done about 50 hours over the last week reading and trying to make it come together but it just won't. Wait for SP1 or Windows 9.

I thought Drive Spaces would be fine since it was going a little modified on their servers but thinking about it servers would be setup with same size and brand server drivers and you probably would not notice.

What I need is large changeable capacity with cheap random drives. WHS 2011 does that and it does it with DrivePool.

Now I just need it on my Windows 7 box and I will be very happy.

muaddib

I am going all in with Windows 8.  The only thing preventing me from doing so would be no Drive Pool availability.  I have been playing with it on a couple of virtual machine and the speed is significantly faster than any of my W7 boxes even with ssd and 16gb ddr3.

It is just around the corner so I hope something is ready prior to its official release.

ian_spencer

Good luck muaddib. I have SSD and 12G Ram and noticed only a bit faster on Windows 8 but seriously they needed to leave the option for Windows 7 desktop. I get Windows 8 free but after 5 days it really is not productive for a desktop, I have flirted with it earlier on and then committed fully for a week and it still does not make sense on a desktop. Drive Pool hopefully is this year. I still have WHS running every few days for backups but power down in between. Seems to be doing well for a stop gap with important stuff backed up online. With Windows Drive Spaces people will discover it's problems probably the hard way as they will think they don't need backups and no one will be able to recover data until tools come available. Waiting for Drive Pool for Windows 7/8, I have tried other options but Drive Pools simplicity and speed seems the best around so my credit card is ready for a couple of licences when it works on Windows 7. 

muaddib

Thanks ian.  I am sorry to hear that you did not enjoy the desktop experience.  Perhaps over time it will grow on you.

Personally I think it is/was a great idea by Microsoft to create one ring to rule them all. 

I agree I learned a long time ago that software raids just doesn't work and calling it storage spaces won't make it work any better.  I've lost about 2tb worth of data and none of the raid reconstruction tools help at all.  Hell I spent about 200 bucks trying to salvage my data.

My option is drive pool or nothing.  I plan on converting my WHS2011 into a W8 server-like PC but another dilemma I am facing besides drive pool compatibility is a network full image backup solution that works similar to WHS2011 backup.  Do you know of any such software?

muaddib

Alex, with Windows 8 just around the corner do you have any news to share? How is version 1.3 progressing? Will it be compatible with W8?

doowruc

It gets worse too. Creating a "Storage Pool" from the control panel is not compatible with the Server Management console, and vice versa. If you create a storage pool from the control panel, you also cannot remove disks unless they are physically attached. However, you can remove disks if you used the Server Management, but it's fairly confusing on how to.  Also, you cannot shrink the pool, only expand it. Also, some of the advanced features (that make it much more reliable) are only available via power shell commands, IIRC. And there is no way to specify a folder is duplicated or not. It's drive level.

Do you know if these issues are still there in the RTM of 2012e?

muaddib

I am not trusting my data to storage spaces, it's drive pool or nothing for me.

Drashna

As far as I'm aware, the issues are "by design". Aka, not to be fixed. I'm not 100% certain, but it looked pretty set. Not to mention, the "control panel" section is what is in the client version of Windows 8. 

The thing that annoys me the most is that other than the device manager, all traces of the disks disappear. Which means no data can be collected from them, IIRC. Which means, no SMART data even. While I know SMART is only a good predictor of mechanical failure, and by no means comprehensive... It was better than nothing. Now you have to hope that Microsoft knows what they're doing. And looking at the rest of the OS gives many doubts about that. Myself included. /rant

Honestly, despite some of the improvements to the OS in general, and to the "Colorado code base", I am not even considering 2012 Essentials. That StableBit doesn't support it yet is just another nail in the coffin.

580guy

DrivePool 1.2

We will have a completely extensible file balancing / file placement plug-in system. It will be extremely simple for anyone with basic programming knowledge to write DrivePool plug-ins that control file placement and balancing.

It will also give every DrivePool user full control over file balancing. You will be able to set limits on which disks can hold duplicated vs. unduplicated files and how much.

The above will be great when using drivepool with flexraid. You will be able to have duplicated shares on 2 drives for smaller files like docs, photos and music and then have unduplicated shares protected by flexraid on the other drives protect by parity (Static content)

This is just what I would like to do.  Duplication is great, but wasteful sometimes.  Currently run FlexRAID + duplication on WHS V1.  Soon to be WHS 2011.  I like all the flexibility of Stablebit, along with the great support and innovation that I see.  Will be jumping in soon!

davidX81

Just want to add my voice in regards to asking about Windows 8 and Windows Server 2012 support... Now that both have been released and Storage Spaces has proved to be a huge flop in relation to problems with multiple disk sizes, parity and lack of balancing, I know I'm really eagerly awaiting DrivePool support with Win Server 2012. Any news or expected release date would be greatly appreciated!

muaddib

I know Alex is hard at work on this but it is really hard waiting.   So far I upgraded 3 of my 7 PC(s).  I plan on upgrading 3 more this weekend with my last PC running WHS2011 waiting on the drivepool update.

Come on Alex the natives are restless, lol.

dvdluvver

Another for support in 2012!   I've begun the upgrade process, but I too have determined that I am NOT trusting my data to StorageSpaces.    Microsoft is getting a lot right these days, but SP is not one of them.   To me, this is a good thing for Alex!   Heck, I'll even pay for an upgrade!

bursal

Hey I 'm another restless native! Drivepool was essential in WHS & I 'm disappointed it's not an option for WS2012E. Like dvdluvver I would pay!
Di

AllPurposeGeek

I am also anxious for Server 2012 Essentials/Storage Essentials support.   I upgraded my home network (I do a lot of testing) to Windows Server 2012 Essentials and Windows Storage Server Essentials R2 as VMs under Hyper-V just so I can still use the stablebit plugin for pooling.  Either a stand-alone app or a dashboard plugin would solve this delima. 

Paaland

Yes please add support for non WHS OS'es as fast as possible. Especially Server 2012 Essentials / Storage Essentials. WHS 2011 is a platform that is quickly dying. I'd really like to upgrade, but I cannot upgrade until DrivePool support is there

muaddib

Join the club.  We are all waiting but unfortunately it looks like all will not be ready until next year, I hope I am wrong.