Multiple pools?

santiagodraco · March 2012

Hello,

I'm interested in purchasing Drive Pool except I need to understand one capability first, if it exists. For me, I'll only use Drive Pool if I can create multiple pools, each with their own drive letter. I want to be able to assign my older slower drives to a pool for less performance oriented data and faster drives to data with higher requirements (ie blueray). Currently there would be no way to assure that blueray content is being written to a faster drive and this could possibly lead to performance glitches if we stream to more than one PC at a time.

I want to move away from RAID except that right now RAID seems to be the only option that will assure me that I'll have the performance I need but if I could have multiple pools I can resolve this issue since I have faster drives that can meet the demands I may run into.

Thanks.

Shane · March 2012

Short answer: as of now DrivePool 1.0 doesn't have multiple pools.

Longer answer: it's been discussed, and if I recall correctly Alex liked the idea of performance-based pooling (if memory serves his preference was still for a single pool but being able to assign individual folders to fast/slow disk sets, don't know if that's changed since M3), but it's not a 1.0 feature. Perhaps you could go with a combination of RAID for the bluray and DrivePool for everything else?

SilverRubicon · March 2012

I can't think of any modern internal drive that is not capable of exceeding the bandwidth of a gigabit ethernet connection. Add them to a pool and apply DP's read striping, and there should be more than enough performance for multiple clients.

santiagodraco · March 2012

Sigh, I think it should be obvious here that the network has nothing to do with what I am talking about. This is about the drives ability to stream multiple Blu-ray files simultaneously. Faster current drives can, older (as in a couple of years older) cannot. I know, I've tried before.

DP does not stripe files it copies them. It adds no more performance to the system than the drives are capable of natively. The only possible performance benefit is when accessing small files that are spread across multiple drives, then you might see some benefit but for large single files, like video, you will not.

Multiple pools that allow you to target bitrate intensive file types on faster drives would be very beneficial to customers looking to build media servers as they can purchase a combination of green and non green drives depending on data requirements.

santiagodraco · March 2012

Short answer: as of now DrivePool 1.0 doesn't have multiple pools.

Longer answer: it's been discussed, and if I recall correctly Alex liked the idea of performance-based pooling (if memory serves his preference was still for a single pool but being able to assign individual folders to fast/slow disk sets, don't know if that's changed since M3), but it's not a 1.0 feature. Perhaps you could go with a combination of RAID for the bluray and DrivePool for everything else?

Thanks Shane. Hopefully they'll announce support for this. For now I'll have to figure something out. I thought about a combination as you described but my whole reason for moving to DP (if I do) is to avoid raid for data integrity reasons. I simply don't trust it long term. I've had drives drop out for example and the raid start a rebuild even though the drive is fine and just needs restarted. I don't want to risk this happening and losing an array (I have one that already I cannot access and I am working on recovery but don't have much hope for success). File duplication is a lot more secure (but of course more expensive hardware wise and provides less performance) and now that it seems that DP is a viable solution I'd like to go that route, but need to solve the issue I mentioned above.

Shane · March 2012

DP does have a read striping option (from the Performance tooltip: "Read striping will read duplicated files from 2 hard drives at the same time") but as you know currently there's no guarantee the two disks will be the fast ones.

Hmm. I imagine your bluray media wouldn't change much. Three possibilities come to mind:

#1: RAID0 your fastest drives, and schedule a nightly differential backup to a pool folder (duplicated or non-duplicated as you prefer)?

#2: DP your fast drives, with a 2nd WHS+DP in a VM with your slow drives in its pool? Might be overkill.

#3: Since DP only balances on write, stop the DP services and access the hidden PoolPart folders to move your bluray files to the two fastest disks and everything else off those disks, then restart the services. This would NOT be officially supported, but it should work. I'd look at scheduling the process with a nightly script, since DP will still save any new pooled files to those disks if they've got more free space than the others!

P.S. Alex has previously talked about making some kind of API available in the future for people who want to write "mods" for DP; if he still plans to do that, #3 could presumably be handled this way also.

datageek · March 2012

Perhaps not something that most users will need, but I can think of a few reasons for wanting multiple pools, including:

differential performance, e.g. fast pool, slow pool as above
large backups, e.g. create a separate pool made up of external drives which can be attached temporarily to support backing up very large (> 3TB) folders
energy saving, e.g. have one "always on" pool for frequently accessed files using drives that are always spun up, and a separate "archive" pool containing files that are rarely accessed, configured to spin these drives down when not in use.

Perhaps we need a separate feature-request forum area...

santiagodraco · March 2012

What datageek said is what I was trying to state in my original post as for the reasons

Great summary.

Performance isolation.

Reduced power consumption.

And separating data types for integrity purposes.

@Shane. Unfortunately I really want to eliminate RAID. Too many issues if a drive goes offline (not fails) and other concerns. I've come to hate it and wish I had drive pooling back

That said I became happy to leave WHS 1's drive pooling because of blueray performance issues and while I expect some improvement here with DP and the fact that it's 2008 R2 based, and the drives are newer, I know that by the nature of how it works some problems will remain. RAID is simply so superior performance wise that it's the only viable solution unless I can assure that specific content goes to specific drives with DP.

I may end up having to try a combination of RAID and DP but I'd really like to avoid that if at all possible.

lildivaaa · April 2012

I too, am finding myself in need of a multiple drive pool solution.

Specifically for the aspect of backups...having drive pool "A" for my primary data, and drive pool "B" for all backups would be wonderful. And I too, am not looking forward to the prospect of RAIDing my backup drives for this purpose.

If DP isn't going to support an ability like this in the future, then I suppose I could build up a bare bones WHS v.1, use my backup drives there and then sync them together.

Or for that matter, maybe build up a FreeNAS server for the specific purpose of backups only.

All that being said.........I'd REALLY rather use DP!!!

Please consider this functionality of multiple pools for a future release.

Philmatic · April 2012

Why not just RAID 1 your drives using Windows Disk Management?

lildivaaa · April 2012

That's a fair question. For me, its a matter of preference...but also, I'd like it to be a "one-stop-shop" solution. I don't like dealing with multiple platforms to accomplish something like this. Kind of like, "the machine is working...why add more machines to the mix? That's just something else that can break down."

We've got a good "machine" with Drive Pool. I'd just like to add a few more "gears" to the DP "machine" for it to be able to do a bit more.

JazJon · April 2012

I have a spare 5 bay eSata enclosure I'd like to use as an off-site backup. It would be nice to have a temporary second pool I can copy all my stuff to once a month. (and then take it to an off-site location) I would do this for fire/theft protection. It would be much easier to have one giant 5 bay hard drive pool, then trying to figure out what will fit where one at a time. I have 9 drives in my main drive pool setup, but my critical stuff could easily fit on the spare 5-bay enclosure. If Alex can make a secondary pool option in the future, I'd use it for what I just described

I tried to create a standard windows spanned disk or raid, but Windows refuses to span/raid disks across USB interfaces

I know DrivePool would work because I test added the drives to my main pool through the USB interface

(I have a 5-port multiplier eSata to USB 3.0/2.0 to adapter) My WHS box only has one eSata port and no slots available to add more.

santiagodraco · April 2012

Alex? Can we get a response to this request? Is it planned? Not planned?

I simply cannot use DP without this functionality or with the ability to target faster drives with specific folders (however I don't see how this is really any different than different pools since you end up having to logically partition out drives anyway).

Alex · April 2012

Here's the way I look at this,

First, DrivePool 1.0 was all about getting a usable and reliable drive pooling solution out the door. The design was always minimalistic in every respect. Every time I wanted to introduce a new feature, I asked, do we really need this? Isn't there a better way to do the same thing, in a simpler way? Sometimes the answer was yes, and sometimes the more complex solution was the only way to go.

Duplication was one of those areas that started very simple but had to grow in complexity over time.

I have thought about multiple pools from day one. DrivePool's architecture was written to support multiple pools, in most areas. It is not 100% code coverage, but it was designed with this in mind.

My feeling right now is that it would add a lot of complexity to DrivePool. The point being, it is one of those features that really requires a thorough point by point review of all the benefits vs. the big negative of having more complexity.

DrivePool's core feature is to provide a single point of storage, it is to get rid of the need to shuffle drives and folders around. Multiple pools seem to counter that.

But, as I've said, I do understand the requirements that were stated above by various people, and am sensitive to those issues and would like to eventually accommodate them.

May I suggest an alternative to multiple pools. If we can provide data separation and performance optimization using a single pool, wouldn't that be better? In the sense that it would keep the existing simplicity of the system while still provide most of the benefits of having multiple pools.

Fast pool vs. slow pool?

This can in theory be accomplished by creating drive groups and then assigning certain folders to certain groups. By default there would be a single drive group and all folders would belong to it.

This way, we don't add an additional step for new users, to "create a pool", which I think is very unintuitive.

But more advanced users can pop open the "drive group" interface and assign drives to groups and folders to groups.

You might even be able to create dynamic drive groups, where files would migrate from one drive to another, given some rules. For example, have a SSD in the "write group". You could copy any files to it really quickly and then have the balancer slowly move those files off the SSD overnight. Since it would all be part of the same pool, your files will continue to be 100% accessible even while this move is in progress. Also, if the SSD fills up, the file placement algorithm would automatically switch to one of the slower drives. Again, since we're on the same pool, this would all just work seamlessly.

How about defining a dynamic group that takes all your hard drives that are older than 3 years old and disallows un-duplicated file placement on them? Again, very trivial to do once you have drive groups.

Detachable pools?

Not sure how WHS would react to this. If a shared folder were "detached" then everything would go haywire. Perhaps this would be a good feature for a future standalone version of DrivePool.

Energy savings?

Could probably be accomplished with drive groups, as described in the first point. Additionally, you can enhance the power savings by using the Scanner to adjust the standby timers on the individual drives that you would want to spin down faster.

I will give this some serious though and see if I can bring the above points into better focus.

Honestly though, I would much rather be working on some much cooler, larger scale features that I have planned for DrivePool

So the short answer is that it's not planned at this time, but I am giving it serous thought as to when / if to put it on the roadmap.

Shane · April 2012

I think from the viewpoint of the end-user, exactly *how* DrivePool achieves the ability to direct certain files to certain disks and whether the developer calls this ability "pools" or "groups" is irrelevant so long as the UI remains elegant and the program does what is wanted. The goal thus becomes creating an "elegant for the programmer" architecture that fits seamlessly into the "elegant for the user" interface.

Re detachable pools: in the context of a DrivePool with support for drive groups, DrivePool would have to be able to stop sharing any shared folders in the group and then remove all drives in the group from the pool as a set without touching their content nor complaining about missing disks (perhaps a new UI task, "Safely remove this drive group from the pool"). Since file duplication should not (IMO) cross group boundaries there should be no risk of the pool losing consistency.

From the user's perspective it should be as simple as plugging the disks back in again afterwards (more complicated from DrivePool's perspective, since it would have to be able to distinguish between the user returning all the disks back, returning only some back, and returning all back but one or more being bad, as well as having to deal with any altered NTFS permissions that might crop up if the disks have been plugged into some other machine in the meantime).

Cooler, larger scale features? Can you give any hints?

saitoh183 · May 2012

The grouping approach is sort of what
http://www.poolit.com/ accomplished with there product

santiagodraco · May 2012

Alex, thanks very much for the reply, sorry it took me so long to see it.

One of the main purposes of WHS is to store media and more and more folks are storing blu-ray, for example, and that requires high consistent data rates. Fast drives are critical, when not using raid and "pooling", but not everyone will want to fill their system with ALL performance drives.   So for some of us this would be an important feature. I hope we see it in a future release.

The implementation methodology, grouping drives vs. multiple pools... really seems to me to be simply a different approach to the same thing. I don't see it changing complexity, really. That said I think the grouping approach is great.   Either solution would work. The goal after all is to be able to assure that certain data is only on certain drives, and that is achieved either way. The other nice thing about the grouping approach you propose is that it means that I, the user, can manage my groups without having to modify which drives are in which pools. I just go the interface and change the "group" and then the magic happens.

But again, I think it's just moving the complexity around. WIth multiple pools I can use the native OS to put my folders on whichever "drive" I want them on. With the grouping you propose, unless you will make a group a drive letter, then I have to start managing folders in DrivePool, so now you need a folder managing interface... more complexity. Multiple pools actually seems simpler in that scenario. Again unless your groups would translate to drive letters to the OS yet there would be a single "pool".

In any case I hope we see this soon.   I really want to start using DrivePool (and I did purchase) but the performance problem with blu-ray is forcing me to keep drive-pool shelved other than as a backup target for my non-media data.

ian_spencer · May 2012

@santiagodraco what format / codec / bit-rate are you ripping blueray to? Is the problem only when multiple people want to play blueray rips or is it giving problems on just one stream?

I was wanting to start putting my DVD and blueray collection on the server so since you know so much about it I would appreciate your comments.

Howdy, Stranger!

Categories

Poll

Multiple pools?

Comments