DrivePool BETA M4 - Migration test

Alex · January 2012

DrivePool BETA M4

DrivePool M4 uses a changed folder structure on the pool. If you're upgrading from M3 you will need to perform an easy migration process from the DrivePool Dashboard tab.

I'd like to test the migration process a bit more widely to make sure that there are no major issues.

If you're upgrading from M3 and would like to be one of these early testers I invite you to download this version of M4:

Download:

http://wiki.covecube.com/StableBit_DrivePool

If you encounter migration issues, send them to stablebit.com/contact.

Instructions:

http://wiki.covecube.com/StableBit_DrivePool_Error_Reports

Alex · January 2012

How to upgrade from M3?

See wiki for instructions:

http://wiki.covecube.com/StableBit_DrivePool

What does migration do?

Here's what DrivePool does to "migrate" the pool to M4

Rename all "ServerPoolPart.G-U-I-D" folders to "PoolPart.G-U-I-D". The GUIDs stay the same.
Set proper NTFS security on each PoolPart folder. Namely, it deletes any custom security applied, and sets the inherit flag to true, to inherit everything from the root.
Remove the ".1" or ".2" suffix from all the folders on the pool and apply a new stream based duplication notation that's not visible in the name.
Unshare all the shares from C:\ServerPool\ServerFolders\...
Re-share all the top level folders on the pool. (The pool drive letter will differ on every system. It can be seen on the DrivePool tab in the Dashboard)
Tell media streaming, HomeGroup and the client computer backup engine that the appropriate folders have moved.
Add the newly shared folders to the Windows Search index.
Remove the old location (if any, just in case) from the Windows Search Index.
Delete any empty folders under C:\ServerPool\...
Normally, all the folders there will be empty at this point.
Non-empty folders are not removed.

Alex · January 2012

What's new

See wiki for full change log:

http://wiki.covecube.com/StableBit_DrivePool

Thank you for testing

ajuhawk · January 2012

Downloading now, I'll post back on how it goes.

phrozen087 · January 2012

I downloaded it also and am giving it a try. Any idea on how long the Background Duplication Pass takes? The total amount of duplicated data I have is around ~9TB so I would expect it wouldn't be instant but was just curious if there was an estimated time.

Aside from that the installation seemed to go smoothly and data still seems to all be intact.

One other question is that a Covecube Virtual Disk 1 that is 2.00 TB in size showed up in the StableBit Scanner. Wasn't exactly sure what this is or if it will just go away on its own at some point.

Editing this to put my transfer notes up as well:

I had 3 main shares (docs, media and video) and all 3 seemed to transfer over properly.
I have accessed some media/files from all 3 shares and video, music and documents all seem to be working well.
The first part, migrate button part took me about 30-40 seconds. I would estimate I have about 750,000 unique files roughly that is about 5TB of unique data on an 18TB chunk of discs.
Ok, duplication appears to have finished checking as my pool
(took a little over an hour) and now says 100%. Weirdly it says that I have 257MB pooled and 257MB duplicated in total. Not exactly sure what that means. I clicked on the remeasure button and will see if it cleans itself up in the next few hours. I'll update this post if anything changes.
That said, all the data is there and accessible and all my shares are there so I am wondering if its just the measuring that is temporarily messed up and it may correct itself on its own, so I will wait and see.

Edit 2:

And that was pretty fast. Took it about 5 minutes to remeasure all of the drives and for the most part everything seems to be completed.
3 minor things here at the end:
There appears to be 687 bytes of data that says "unduplicated" on one of my 2TB drive
The virtual disk is still showing up in Stablebit Scanner
When I add up the the free space and the total used space they don't equal the total space available for the pool. It is about 160GB off in the calculation. Not really a big issue, but another odd point.

ajuhawk · January 2012

Upgrade Notes:

Installed new file, immediately rebooted upon completion.
Opened Dashboard and clicked the migrate button (this process took awhile, ~5TB of data on 10TB of space -- maybe 10 mins, so not bad really.
Noted new drive letter with all data accessible
Only 1 share got correctly transferred (client computer backups)
Started looking on how to fix the other shares...
decided to reboot again...didn't help, may be time to contact Alex...

Doug · January 2012

Upgraded with no issues. Same steps a "ajuhawk".

But I had no issues with shares: 14.6TB and 12 shares

About 15 minutes for duplication pass and full measure pass. but still Indexing 45 minutes later. But I figure 14.6TB of data will take awhile.

Remote Web Access Media Library is all jacked up again, will have to see what it looks like after Indexing finishes.

Edit 1- almost 2hrs still indexing- Remote Web Access Media Library numbers are changing with indexing.

phrozen087 · January 2012

Upgraded with no issues. Same steps a "ajuhawk".
But I had no issues with shares: 14.6TB and 12 sharesAbout 15 minutes for duplication pass and full measure pass. but still Indexing 45 minutes later. But I figure 14.6TB of data will take awhile. Remote Web Access Media Library is all jacked up again, will have to see what it looks like after Indexing finishes.

When you finish could you see if your numbers in the Drivepool tab add up to the correct total amount? The number listed as the total for each share + your free space and see if it equals the total listed up top.

ajuhawk · January 2012

2/3, not too bad

It's weird, I think the "links" just didn't get made correctly and I'm not sure where those are to try and fix them...

phrozen087 · January 2012

I figured out the discrepancy in disk space. One of the disks in my pool has a couple hundred GBs of data that isn't stored in drive pool as a backup of several files. I am pretty sure that the overall size and freespace take the actual physical disk sizes into account and not the space available only to the pool. This accounts for the missing space.

Only thing I still have to try and figure out is why I have 692 bytes of unduplicated data on a drive that contains only drivepool information.

Doug · January 2012

@phrozen087

There is a slight calculation difference between Windows (1.81) and Drivepool (1.82)

ajuhawk · January 2012

Doug, phrozen087,

On the new drive letter that appears, do you see your shared directories right on the root or are they in a sub-folder? mine are on the root of the pool drive.

Symo · January 2012

Excellent timing! M3 has started causing some major issues for me - all sorts of different services that DP interacts with keep crashing, which keeps logging out the Administrator account, the drivepool sometimes becomes unavailable, if I've got something streaming over twonky that cuts out... So hopefully M4 will fix all that!

Will be trying this out straight away when I get home tonight!

phrozen087 · January 2012

Doug, phrozen087,

On the new drive letter that appears, do you see your shared directories right on the root or are they in a sub-folder? mine are on the root of the pool drive.

On the new drive (looks to be the big virtual disc that drivepool created) I see the shares right in the root with little locks on the folder icons. The actual data is still in the hidden folders that are named PoolPart.##### <- long identifier and then its all down in subfolders on those.

Now that my system has sat for a few hours everything seems to have cleared up except for the fact that 687 bytes of data say there are "Unduplicated files" on one of my hard discs. The drives all say healthy though (I would think they would say unhealthy if something wasn't duplicated) so I am not sure what this is, maybe it will clear up eventually also on its own.

ajuhawk · January 2012

On the new drive (looks to be the big virtual disc that drivepool created) I see the shares right in the root with little locks on the folder icons. The actual data is still in the hidden folders that are named PoolPart.##### <- long identifier and then its all down in subfolders on those.

Okay, thanks. When you create a new share it wants to put them in the big virtual disk\serverfolder\newShare, so I wasn't sure which was the default for the migration.

Anyway, looks like with some registry editing/hacking I can get the shares to work again... not sure which part of my migration didn't work...

JazJon · January 2012

I'm all excited to migrate from M3 to M4, but the repair of my unhealthy M3 drives is taking forever! I'm on my 3rd of 4th unhealthy drive repair, started 5 hours ago. Go figure the longest repair time I've ever had is the day I want to update to M4. Time for bed, I'll try again tomorrow after 4 out of 4 is repaired. Hopefully constant unhealthy/unduplicated drives will be a thing of the past as soon as I'm on the all new M4! I'm sure it was worth the wait and I'll report in my how mine went. 18TB, maybe 10TB full right now.

Saltydh · January 2012

All looking good from here, 15 TB migrated and is currently indexing. I've added a new drive to the pool and that all went nice and smoothly.

One warning was displayed during duplication processing. It indicated that I had duplicate files. I selected the check box to delete the oldest duplicate file. Processing is now complete (100%) however my pool statistics indicate 8.41 MB (yeah meg) of duplication. As I'm not actually using the duplication features of DrivePool, I'm assuming these are files that are causing a collision because of the file path. At some stage in the past I've probably manually added files to a drive's pool folder.

Not sure if I should have waited until the duplication processing was complete before resolving the warning. Either way I'm curious to know if there's any way of querying Drive Pool to ascertain the location of the duplicate files so I can manually remove them??

Very very nice work on what's shaping up to be a fantastic M4 release. I tried to hold back but I just can't help myself. This is sooo much cooler than Storage Spaces! I'm also assuming that it's a coincidence that M$ is just M4 with an extra Shift key ;P

Alex · January 2012

I downloaded it also and am giving it a try. Any idea on how long the Background Duplication Pass takes? The total amount of duplicated data I have is around ~9TB so I would expect it wouldn't be instant but was just curious if there was an estimated time.

...

One other question is that a Covecube Virtual Disk 1 that is 2.00 TB in size showed up in the StableBit Scanner. Wasn't exactly sure what this is or if it will just go away on its own at some point.

...

On a perfectly duplicated pool, the pass just enumerates all the files to make sure they're duplicated correctly. It should be pretty fast.

The Scanner sees the "virtual" physical disk. I realize that what I just said is a contradiction. 2.0 TB is the fixed size of the emulated physical disk that the pool is on. Since there really is no physical disk, this size is of no consequence.

The Scanner will be updated eventually to recognize this disk as a DrivePool disk.

Alex · January 2012

Edit 1- almost 2hrs still indexing- Remote Web Access Media Library numbers are changing with indexing.

Indexing is a "Windows Search" process. It is affected by how "busy" it thinks your computer is. DrivePool has no control over the indexing speed.

WHS always indexes everything that's shared, DrivePool just shows the status.

Indexing is not required for the pool to function. DrivePool shows you a pie chart based on the Windows Search index when you select a folder.

Alex · January 2012

Now that my system has sat for a few hours everything seems to have cleared up except for the fact that 687 bytes of data say there are "Unduplicated files" on one of my hard discs. The drives all say healthy though (I would think they would say unhealthy if something wasn't duplicated) so I am not sure what this is, maybe it will clear up eventually also on its own.

Windows tends to put some metadata on the disk, in a non-shared folder. Anything placed on the pool in a folder that's not duplicated is considered un-duplicated data. This is probably what you're seeing.

The recycling bin is also un-duplicated, if that's enabled.

Alex · January 2012

IMPORTANT

Thank you all for testing early.

I've pulled the link for now because there's an issue with file renames. It's a very specific issue where if your pool is balanced equally across all drives, and something tries to rename a file while asking to overwrite the destination, there is a chance that the destination will not be overwritten properly.

Again, this is pretty specific, but the WHS backup engine renames files often so this needs to be fixed in order to maintain backup integrity.

I'll have this fixed up ASAP (1 to 2 days I think) and put up a new build.

Paaland · January 2012

Just managed to get it then

Migration went fine btw. And it's now indexing all the shares.

Alex · January 2012

Not sure if I should have waited until the duplication processing was complete before resolving the warning. Either way I'm curious to know if there's any way of querying Drive Pool to ascertain the location of the duplicate files so I can manually remove them??

Perform a "re-measure" from the DrivePool tab. If the duplicated usage doesn't go away after that finishes, then you must have a duplicated folder somewhere. (If you're sure that you don't then it's a bug)

DrivePool doesn't know exactly what files are duplicated. For efficiency, it doesn't keep that sort of fine grained statistics (that would require a database).

CoveFS monitors your pool for duplication issues in real-time and if it detects a problem, it notifies the DrivePool service which performs a full duplication pass over the entire pool.

A duplication pass will clean up files that are duplicated with too many file parts, so that's why I said that if you see space used by duplication, something must actually be in a duplicated folder somewhere.

Sg112 · January 2012

I migrate to M4 and update it went very well. Then I got a home server back up error a few hours latter; all back ups have errors. I tryed to repair but this does not work so I delete all back ups. Then I have 2 harddrives fail in about 2 hours. I put these into another server, but they are still the same both drive cause windows to stop responding for a long time. Can it be possible M4 cause damage to harddrives?

Saltydh · January 2012

Thanks Alex,

I tracked down the duplicates after the re-measure produced the same result. No bugs to report here, everything came good after the duplicates were manually removed.

Cheers!

letho · January 2012

Hey, i have not yet set up M3 on my whs, so i cannot migrate ;-) Can i set up m4 for testing now, or should i wait until the current issues are solved?

And one question, can i rename the drive letter?

Edit:

Ok, i have installed it :-) it seems to work and i have already renamed the pool drive letter. now i'm zrying to add the disk to the pool.

phrozen087 · January 2012

Thanks Alex,

I tracked down the duplicates after the re-measure produced the same result. No bugs to report here, everything came good after the duplicates were manually removed.

Cheers!

How did you find the duplicates manually? I was wondering if I could do something similar to find whatever this one tiny unduplicated file is on my pool (its this annoying 697 byte file, but it annoys me!)

In other news, after letting everything set over night I came back this morning and everything appears to be running well. Read/Write rates to the pool are also much faster then before, which is always a good thing.

I also found one additional minor bug: If the discs are asleep (I have my discs auto spin down after 10 mins if I am not using them) and I load the dashboard up fresh then Drivepool says I have no pool or discs in the pool. If I go access a disc or share, then the pool wakes up and if I reload the dashboard everything is normal. Doesn't really affect anything, but appears to be a bug. Also for reference, not sure if this has anything to do with it, my pool is 8 discs connected via eSata with 4 discs each inside 2 MediaSonic 4 bay enclosures. This behavior didn't occur in M3 ever.

Saltydh · January 2012

Phrozen087: How did you find the duplicates manually?

It's a corporate dedupe product I ran across the drives (in house stuff). Free alternative, NoDupe is something I've used before in a pinch. (MP3s and the like). http://www.otissoft.co.uk/products/index.html

Direct download link because Brothersoft is horrible:http://otissoft.co.uk/products/nodupe/bin/NoDupe_1_16_7_2.exe

Hope it helps.

dvdluvver · January 2012

Just curious - are the bugs Alex is currently working out (that caused him to remove the M4 link) strictly migration related? I only ask because I have been sitting on the sidelines awaiting M4 to take the plunge with my WHS! Can't wait to get this going! My old WHSv1 (which I am using as my "duplication" system is starting to get pretty cranky.

Shane · January 2012

The one Alex mentioned as the reason for him pulling the download link was not migration related. http://forum.covecube.com/discussion/comment/1517#Comment_1517

JazJon · January 2012

Just curious - are the bugs Alex is currently working out (that caused him to remove the M4 link) strictly migration related? I only ask because I have been sitting on the sidelines awaiting M4 to take the plunge with my WHS! Can't wait to get this going! My old WHSv1 (which I am using as my "duplication" system is starting to get pretty cranky.

Alex said:
"I've pulled the link for now because there's an issue with file renames. It's a very specific issue where if your pool is balanced equally across all drives, and something tries to rename a file while asking to overwrite the destination, there is a chance that the destination will not be overwritten properly.
Again, this is pretty specific, but the WHS backup engine renames files often so this needs to be fixed in order to maintain backup integrity.
I'll have this fixed up ASAP (1 to 2 days I think) and put up a new build."

I'm finally ready to update M3 to M4 as soon as the fixed version comes out

The repair of M3 unduplicated files took me 2 days to complete.

Howdy, Stranger!

Categories

Poll

DrivePool BETA M4 - Migration test

Comments