Status of BETA M3

sean10780 · July 2011

Yes Shane, seems if I had a new file to the drives, it makes it unhealthy, and then there is certain new files to cause it to go unhealthy over and over again.

Shane · July 2011

To clarify, if you create - right now - a new pooled, duplicate-enabled share, then create a text file in it, and then modify that text file, does the drive become unhealthy? And if (after repairing) you repeat with a non-duplicate-enabled share, does the drive become unhealthy?

Hmm. While my personal suspicion is NTFS ACL issues from DE, I could be completely off-base, so it might be best if both of you could submit a log file to Alex and he can get to the heart of the problem. http://wiki.covecube.com/StableBit_DrivePool_Log_Collection

Alex · July 2011

Status update on M3 as of today (7/11/11):

Everything mentioned in the last update is coded. There's going to be one more thing that will be added and that's data visualization. It's already in progress and will be completed today or tomorrow.

Once that's done, everything will need to be tested and any issues will need to be fixed up. Then it goes public.

rubenhak · July 2011

Alex, Can you explain the main workflow of DrivePool.

I just logged in to my server and noticed excessive CPU consumption by DrivePool service. In ResourceMonitor I also noticed that it was accessing the pool contents. However those files were not modified after pool creation. Please explain what was the service actually doing.

Thank you!

Stacy · July 2011

Hi Alex,

Long time WHS V1 user and reseller, have to say I think your product looks like a winner to me.
I am eager as is everyone else seemingly on your forum to hear you say it is stable and ready for main stream use.

Curious as to how big of a pool has been attempted with your product at this point, and across how many drives.

Best Regards,

Stacy Laboucane
Internet Guys Inc.

Alex · July 2011

M3 Screen: http://i.imgur.com/64Mui.png

From twitter.

Alex · July 2011

...
I just logged in to my server and noticed excessive CPU consumption by DrivePool service. In ResourceMonitor I also noticed that it was accessing the pool contents. However those files were not modified after pool creation. Please explain what was the service actually doing.

Thank you!

The job of the DrivePool service is to access the pool

In M2 the service does nothing except service I/O requests to the pool. Nothing else.

CPU usage is caused by something accessing pooled files / directories.

Of course if you suspect something going wrong then report it using http://stablebit.com/contact

M4 will aim to bypass the service for CPU consuming I/O tasks, thus speeding everything up.

---

M3 is more complicated and that's a different discussion. I will post more info when it's out.

Alex · July 2011

Hi Alex,

Long time WHS V1 user and reseller, have to say I think your product looks like a winner to me.
I am eager as is everyone else seemingly on your forum to hear you say it is stable and ready for main stream use.

Curious as to how big of a pool has been attempted with your product at this point, and across how many drives.

Best Regards,

Stacy Laboucane
Internet Guys Inc.

Stacy,

Thanks for your support.

Everything is moving as quickly as possible while trying not to compromise the design

As far as pool size, I'm sure some of the beta testers on here can claim the pool size crown. Internally DrivePool testing is concentrating on interface diversity not total size. SATA / SSD / USB 2.0 / 3.0 are used for testing. The pool size is only a few terabytes.

I have to say, I'm pretty excited about M3, once it's stabilized I think it will have all the core features that people have been asking for.

I think we can look forward to testing with 16 TB+ once it's ready.

rubenhak · July 2011

Alex, I understand that service was accessing the pool which introduces CPU consumption, that's totally normal, but I'm curious why does it access those files. I'm 100% positive that nothing I'm my network was accessing any on shares on the server.

Thanks, Ruben

Also, when is M3 scheduled? It looks awesome

sean10780 · July 2011

M3 does look pretty good, and I'm looking forward to it.

Danee · July 2011

M3 looks very good! Great job!

Stacy · July 2011

Stacy,

Thanks for your support.

Everything is moving as quickly as possible while trying not to compromise the design

As far as pool size, I'm sure some of the beta testers on here can claim the pool size crown. Internally DrivePool testing is concentrating on interface diversity not total size. SATA / SSD / USB 2.0 / 3.0 are used for testing. The pool size is only a few terabytes.

I have to say, I'm pretty excited about M3, once it's stabilized I think it will have all the core features that people have been asking for.

I think we can look forward to testing with 16 TB+ once it's ready.

Hi Alex,

Thanks for taking the time and making the effort to address such a screw up as MS dropping the DE. In reality it looks like your product will be superior so that is awesome and you turned it into a business opperunity too boot ..Kudos to you

I currently run 2 WHS V1 machines with 17TB, (one is a synctoy back up of the first) and have several clients with similar setups. One drawback has always been that V1 did not support the new SATA6 drives, so we have had to attach the faster drives to slower SATA ports for them to work.

So when they annouced WHS v2.0 I was very excited to see it was on 64bit and would see the large, faster drives natively. Then they dropped DE ! Ack ! What a rollercoaster ride...

At any rate seems we are on an up swing now thanks to you and your product.

Keep up the good work !

Stacy Laboucane
Internet Guys Inc.

Alex · July 2011

Great news. M3 is now being finalized and should be released within days.

Here are some of the new features:

DrivePool folders are now recognized by WHS as regular folders. This means that DrivePool folders will work with media streaming and remote web access just like any other folder.
Enable / disable pooled folder duplication without recreating the folder.
Migrate any folder to and from the pool, including the default WHS folders.
Data visualization. See folder usage, drive usage, and pool usage visually using pie charts.
File balancing. Efficient file balancing in order to ensure maximum usage of all available disk space.

There's really a lot more to say here, especially about file balancing. I'll post more info once M3 is out.

Alex · July 2011

Alex, I understand that service was accessing the pool which introduces CPU consumption, that's totally normal, but I'm curious why does it access those files. I'm 100% positive that nothing I'm my network was accessing any on shares on the server.

Thanks, Ruben

Also, when is M3 scheduled? It looks awesome

The M2 service should not access anything on its own. It's essentially acting like a proxy for the pooled drive. You might want to try using Process Monitor to see exactly what files are being accessed.

One exception is when you perform tasks. Like removing a drive from the pool or integrating a foreign disk. In those cases while service is performing the task asynchronously it will be accessing files on the pool. But tasks will terminate once they're complete or there's a problem encountered.

Once a task is started, it does not require user interaction and will continue to run regardless of whether the Dashboard is active or not.

Alex · July 2011

Hi Alex,

Thanks for taking the time and making the effort to address such a screw up as MS dropping the DE. In reality it looks like your product will be superior so that is awesome and you turned it into a business opperunity too boot ..Kudos to you

I currently run 2 WHS V1 machines with 17TB, (one is a synctoy back up of the first) and have several clients with similar setups. One drawback has always been that V1 did not support the new SATA6 drives, so we have had to attach the faster drives to slower SATA ports for them to work.

So when they annouced WHS v2.0 I was very excited to see it was on 64bit and would see the large, faster drives natively. Then they dropped DE ! Ack ! What a rollercoaster ride...

At any rate seems we are on an up swing now thanks to you and your product.

Keep up the good work !

Stacy Laboucane
Internet Guys Inc.

Thank you. I hope the final version will live up to everyone's expectations.

The current BETA M2, while functional, visually looks a bit incomplete. BETA M3 should fill in the gaps in terms of functionality and missing UI.

I can't wait to get BETA M4 out so that the original vision for DrivePool is complete! It has been a challenge making DrivePool a reality, but I hope that some time in August we will have a working BETA M4. After that it's release time.

Alex · July 2011

Ok, I had to post this:

Working out the kinks

wordgasm · July 2011

Well this is drivebender but I am checking out both.

Bet I have more drive space than you!

https://picasaweb.google.com/102122954229684497539/PBs?authuser=0&authkey=Gv1sRgCNyFjbHcuNvpgAE&feat=directlink

rubenhak · July 2011

Alex, I understand that service was accessing the pool which introduces CPU consumption, that's totally normal, but I'm curious why does it access those files. I'm 100% positive that nothing I'm my network was accessing any on shares on the server.

Thanks, Ruben

Also, when is M3 scheduled? It looks awesome
The M2 service should not access anything on its own. It's essentially acting like a proxy for the pooled drive. You might want to try using Process Monitor to see exactly what files are being accessed.

One exception is when you perform tasks. Like removing a drive from the pool or integrating a foreign disk. In those cases while service is performing the task asynchronously it will be accessing files on the pool. But tasks will terminate once they're complete or there's a problem encountered.

Once a task is started, it does not require user interaction and will continue to run regardless of whether the Dashboard is active or not.

Yeah, actually that was my initial question. I'd like to get more familiar about those tasks that DrivePool performs. Are you planning to document it and have it available to public?

P.S. Is that a screenshot from youtube or google datacenter using DrivePool?

Alex · July 2011

Yeah, actually that was my initial question. I'd like to get more familiar about those tasks that DrivePool performs. Are you planning to document it and have it available to public?

P.S. Is that a screenshot from youtube or google datacenter using DrivePool?

If I understand your original question correctly, you are asking what tasks DrivePool performs automatically in the background?

The answer to that is none, in BETA M2.

In M2 all the tasks are started from the Dashboard GUI and are documented in the various blog posts (especially the last 2).

See: http://wiki.covecube.com/StableBit_DrivePool#Blog_posts

To sum it up, every wizard is a service task. DrivePool does as little as possible in the UI and hands off all work to the background service. Keep in mind, these are manually triggered by the user.

The most obvious one is the Remove drive wizard.

Alex · July 2011

P.S. Is that a screenshot from youtube or google datacenter using DrivePool?

Yes, Google called us and they want to make a backup of the Internet... Just in case

rubenhak · July 2011

Alex, I got another question. Lets say I shutdown/restart the system. How does DrivePool determine healthy status of my pool? I mean by the time DrivePool was turned off, some of duplicated files could get deleted/damaged?

Will DrivePool try to rescan all files under pool and mark those files as unhealthy even if user don't access the pool at all?

Shane · July 2011

Not Alex, but if it's a normal (i.e. requested) shutdown/restart, Windows says "hey, all processes, shutting down, please finish what you're doing and let me know", and DrivePool says, "okay... alright, finished up here, it's safe to turn me off now".

If on the other hand it's a BSOD or you've tripped over the power cord, it'll depend on DrivePool's ability to keep track of its state. "Hmm, okay, user wants me to copy file A to folder B, making a note... okay, finished that, making a note... okay, another file to copy, making a note... *Argh! BSOD*... (system reboots, DrivePool starts up) okay looks like I didn't shutdown properly, let me check my notes..."

Shane · July 2011

@wordgasm - holy batdrives batman... 35.9 PB?! That's some serious hardware! For real?

(what's even scarier is the "2.80 TB free" in context of the above)

Alex · July 2011

Alex, I got another question. Lets say I shutdown/restart the system. How does DrivePool determine healthy status of my pool? I mean by the time DrivePool was turned off, some of duplicated files could get deleted/damaged?
Will DrivePool try to rescan all files under pool and mark those files as unhealthy even if user don't
access the pool at all?

DrivePool does not actively scan the entire pool for consistency.

Every time you open a duplicated file DrivePool performs a quick sanity check on the file to make sure that all the duplicated parts are consistent. If the file is not consistent then it's tagged with some external metadata (not on the file, in C:\ProgramData\...) marking the drive and the particular file suspect.

So if you were to shut down, damage some duplicated files and restart, DrivePool would detect the problem at the time that the damaged duplicated file would be accessed. Since it's a duplicated file the access would succeed, but the file would be tagged as damaged. It would be re-duplicated with the good part after running the repair wizard.

This is similar to what would happen on a non-pooled volume if a sector went bad while the system was turned off. You wouldn't know it until you accessed the file.

---

Just to comment on what Shane added. Yes, DrivePool does have a safe shut-down sequence that unmounts the pool safely and shuts down all the internal "Services". The main service is modular and it contains a suite of mini-service-like modules each interacting with each other, so that it's easy to extend the code and add new functionality.

In the event of a power cord pull, the worst case scenario that I can think of is that you were in the middle of writing a duplicated file. What would happen is the NTFS metadata could get corrupted, but because NTFS is a journaling file system, it would repair itself. NTFS guarantees the non-corruption of its metadata (not the files on it). DrivePool would have a corrupt duplicated file, but it wouldn't matter much because any file that was being written as the system powered off is most likely unusable. It would be tagged corrupt as described above, but you would almost certainly just delete it and start over.

rubenhak · July 2011

Alex, do you think you expose a feature to have an ability to scan entire pool/folder/file for consistency upon user request? This might be a good feature to keep your data safe. Otherwise your hdd can go bad, and your data can get corrupted without you even knowing it (if you don't access corrupted files).

dvdluvver · July 2011

Hi, Alex - it's been awhile!

I have two questions:

1 - M3 is close, right?

2 - when removing a drive from the pool, do the shares remain available? I remember a big pain with WHS v1 was that if you wanted to remove a drive, the whole server was essentially taken offline (which was not appreciated by my family!)

I have my old WHS v1 serving as a backup of all my data (some 8TB worth), so I'm going to try M3 in my "live" WHS 2011 server (with some duplicated folders and some not, knowing that if everything crashes I can retrieve my backups) to see how she flies. Looks like it's really coming together nicely (and my suspicion is that this is going to be the most desired option for DE replacement for the masses!)!

Alex · July 2011

Alex, do you think you expose a feature to have an ability to scan entire pool/folder/file for consistency upon user request? This might be a good feature to keep your data safe. Otherwise your hdd can go bad, and your data can get corrupted without you even knowing it (if you don't access corrupted files).

That's a big question, perhaps we should start a new discussion on this point.

Alex · July 2011

Hi, Alex - it's been awhile!

I have two questions:

1 - M3 is close, right?
2 - when removing a drive from the pool, do the shares remain available? I remember a big pain with WHS v1 was that if you wanted to remove a drive, the whole server was essentially taken offline (which was not appreciated by my family!)

I have my old WHS v1 serving as a backup of all my data (some 8TB worth), so I'm going to try M3 in my "live" WHS 2011 server (with some duplicated folders and some not, knowing that if everything crashes I can retrieve my backups) to see how she flies. Looks like it's really coming together nicely (and my suspicion is that this is going to be the most desired option for DE replacement for the masses!)!

The binary is ready to go, it's 2AM here, so Tomorrow
When removing a drive from the pool the entire pool is not taken offline. All the unduplicated files on the drive being removed become inaccessible until the removal process is complete. Duplicated files remain accessible but may not have duplication until removal is complete.

As a followup to 2, now that M3 is complete, I should add that when balancing folders the system moves files around. As a result it will lock an individual file for the duration of the file move and then unlock it. If a file cannot be locked because something has it open then it's not moved in the balancing pass.

Balancing is only performed when necessary. Necessity is defined by a formula based on data coming from the pool in real-time. In other words, no balancing every 30 minutes. Automatic balancing is limited to every 24 hours, when needed, and only at night. Manual balancing can be started at any time from the UI.

mqualizza · July 2011

While we wait for M3 to be posted.........Alex can you give us an idea on cost to purchase the Ad-on?

danivtec · July 2011

I recall an earlier figure of $20 in the early development releases.

Howdy, Stranger!

Categories

Poll

Status of BETA M3

Comments