DrivePool BETA M4 - Migration test

Alex

DrivePool BETA M4

DrivePool M4 uses a changed folder structure on the pool. If you're upgrading from M3 you will need to perform an easy migration process from the DrivePool Dashboard tab.

I'd like to test the migration process a bit more widely to make sure that there are no major issues.

If you're upgrading from M3 and would like to be one of these early testers I invite you to download this version of M4:

Download:

http://wiki.covecube.com/StableBit_DrivePool

If you encounter migration issues, send them to stablebit.com/contact.

Instructions:

http://wiki.covecube.com/StableBit_DrivePool_Error_Reports

Alex

How to upgrade from M3?

See wiki for instructions:

http://wiki.covecube.com/StableBit_DrivePool

What does migration do?

Here's what DrivePool does to "migrate" the pool to M4

Rename all "ServerPoolPart.G-U-I-D" folders to "PoolPart.G-U-I-D". The GUIDs stay the same.
Set proper NTFS security on each PoolPart folder. Namely, it deletes any custom security applied, and sets the inherit flag to true, to inherit everything from the root.
Remove the ".1" or ".2" suffix from all the folders on the pool and apply a new stream based duplication notation that's not visible in the name.
Unshare all the shares from C:\ServerPool\ServerFolders\...
Re-share all the top level folders on the pool. (The pool drive letter will differ on every system. It can be seen on the DrivePool tab in the Dashboard)
Tell media streaming, HomeGroup and the client computer backup engine that the appropriate folders have moved.
Add the newly shared folders to the Windows Search index.
Remove the old location (if any, just in case) from the Windows Search Index.
Delete any empty folders under C:\ServerPool\...
Normally, all the folders there will be empty at this point.
Non-empty folders are not removed.

Alex

What's new

See wiki for full change log:

http://wiki.covecube.com/StableBit_DrivePool

Thank you for testing :)

ajuhawk

Downloading now, I'll post back on how it goes.

phrozen087

I downloaded it also and am giving it a try. Any idea on how long the Background Duplication Pass takes? The total amount of duplicated data I have is around ~9TB so I would expect it wouldn't be instant but was just curious if there was an estimated time.

Aside from that the installation seemed to go smoothly and data still seems to all be intact.

One other question is that a Covecube Virtual Disk 1 that is 2.00 TB in size showed up in the StableBit Scanner. Wasn't exactly sure what this is or if it will just go away on its own at some point.

Editing this to put my transfer notes up as well:

I had 3 main shares (docs, media and video) and all 3 seemed to transfer over properly.
I have accessed some media/files from all 3 shares and video, music and documents all seem to be working well.
The first part, migrate button part took me about 30-40 seconds. I would estimate I have about 750,000 unique files roughly that is about 5TB of unique data on an 18TB chunk of discs.
Ok, duplication appears to have finished checking as my pool
(took a little over an hour) and now says 100%. Weirdly it says that I have 257MB pooled and 257MB duplicated in total. Not exactly sure what that means. I clicked on the remeasure button and will see if it cleans itself up in the next few hours. I'll update this post if anything changes.
That said, all the data is there and accessible and all my shares are there so I am wondering if its just the measuring that is temporarily messed up and it may correct itself on its own, so I will wait and see.

Edit 2:

And that was pretty fast. Took it about 5 minutes to remeasure all of the drives and for the most part everything seems to be completed.
3 minor things here at the end:
There appears to be 687 bytes of data that says "unduplicated" on one of my 2TB drive
The virtual disk is still showing up in Stablebit Scanner
When I add up the the free space and the total used space they don't equal the total space available for the pool. It is about 160GB off in the calculation. Not really a big issue, but another odd point.

ajuhawk

Upgrade Notes:

Installed new file, immediately rebooted upon completion.
Opened Dashboard and clicked the migrate button (this process took awhile, ~5TB of data on 10TB of space -- maybe 10 mins, so not bad really.
Noted new drive letter with all data accessible
Only 1 share got correctly transferred (client computer backups)
Started looking on how to fix the other shares...
decided to reboot again...didn't help, may be time to contact Alex...

Doug

Upgraded with no issues. Same steps a "ajuhawk".

But I had no issues with shares: 14.6TB and 12 shares

About 15 minutes for duplication pass and full measure pass. but still Indexing 45 minutes later. But I figure 14.6TB of data will take awhile.

Remote Web Access Media Library is all jacked up again, will have to see what it looks like after Indexing finishes.

Edit 1- almost 2hrs still indexing- Remote Web Access Media Library numbers are changing with indexing.

phrozen087

Doug said:
Upgraded with no issues. Same steps a "ajuhawk".
But I had no issues with shares: 14.6TB and 12 shares
About 15 minutes for duplication pass and full measure pass. but still Indexing 45 minutes later. But I figure 14.6TB of data will take awhile.
Remote Web Access Media Library is all jacked up again, will have to see what it looks like after Indexing finishes.

When you finish could you see if your numbers in the Drivepool tab add up to the correct total amount? The number listed as the total for each share + your free space and see if it equals the total listed up top.

ajuhawk

2/3, not too bad ;)

It's weird, I think the "links" just didn't get made correctly and I'm not sure where those are to try and fix them...

phrozen087

I figured out the discrepancy in disk space. One of the disks in my pool has a couple hundred GBs of data that isn't stored in drive pool as a backup of several files. I am pretty sure that the overall size and freespace take the actual physical disk sizes into account and not the space available only to the pool. This accounts for the missing space.

Only thing I still have to try and figure out is why I have 692 bytes of unduplicated data on a drive that contains only drivepool information.

Doug

@phrozen087

There is a slight calculation difference between Windows (1.81) and Drivepool (1.82)

ajuhawk

Doug, phrozen087,

On the new drive letter that appears, do you see your shared directories right on the root or are they in a sub-folder? mine are on the root of the pool drive.

Symo

Excellent timing! M3 has started causing some major issues for me - all sorts of different services that DP interacts with keep crashing, which keeps logging out the Administrator account, the drivepool sometimes becomes unavailable, if I've got something streaming over twonky that cuts out... So hopefully M4 will fix all that!

Will be trying this out straight away when I get home tonight!

phrozen087

ajuhawk said:
Doug, phrozen087,

On the new drive letter that appears, do you see your shared directories right on the root or are they in a sub-folder? mine are on the root of the pool drive.

On the new drive (looks to be the big virtual disc that drivepool created) I see the shares right in the root with little locks on the folder icons. The actual data is still in the hidden folders that are named PoolPart.##### <- long identifier and then its all down in subfolders on those.

Now that my system has sat for a few hours everything seems to have cleared up except for the fact that 687 bytes of data say there are "Unduplicated files" on one of my hard discs. The drives all say healthy though (I would think they would say unhealthy if something wasn't duplicated) so I am not sure what this is, maybe it will clear up eventually also on its own.

ajuhawk

phrozen087 said:

ajuhawk said:
On the new drive (looks to be the big virtual disc that drivepool created) I see the shares right in the root with little locks on the folder icons. The actual data is still in the hidden folders that are named PoolPart.##### <- long identifier and then its all down in subfolders on those.

Okay, thanks. When you create a new share it wants to put them in the big virtual disk\serverfolder\newShare, so I wasn't sure which was the default for the migration.

Anyway, looks like with some registry editing/hacking I can get the shares to work again... not sure which part of my migration didn't work...

JazJon

I'm all excited to migrate from M3 to M4, but the repair of my unhealthy M3 drives is taking forever! I'm on my 3rd of 4th unhealthy drive repair, started 5 hours ago. Go figure the longest repair time I've ever had is the day I want to update to M4. Time for bed, I'll try again tomorrow after 4 out of 4 is repaired. Hopefully constant unhealthy/unduplicated drives will be a thing of the past as soon as I'm on the all new M4! I'm sure it was worth the wait and I'll report in my how mine went. 18TB, maybe 10TB full right now.

Saltydh

All looking good from here, 15 TB migrated and is currently indexing. I've added a new drive to the pool and that all went nice and smoothly.

One warning was displayed during duplication processing. It indicated that I had duplicate files. I selected the check box to delete the oldest duplicate file. Processing is now complete (100%) however my pool statistics indicate 8.41 MB (yeah meg) of duplication. As I'm not actually using the duplication features of DrivePool, I'm assuming these are files that are causing a collision because of the file path. At some stage in the past I've probably manually added files to a drive's pool folder.

Not sure if I should have waited until the duplication processing was complete before resolving the warning. Either way I'm curious to know if there's any way of querying Drive Pool to ascertain the location of the duplicate files so I can manually remove them?? :D

Very very nice work on what's shaping up to be a fantastic M4 release. I tried to hold back but I just can't help myself. This is sooo much cooler than Storage Spaces! I'm also assuming that it's a coincidence that M$ is just M4 with an extra Shift key ;P

Alex

phrozen087 said:
I downloaded it also and am giving it a try. Any idea on how long the Background Duplication Pass takes? The total amount of duplicated data I have is around ~9TB so I would expect it wouldn't be instant but was just curious if there was an estimated time.

...

One other question is that a Covecube Virtual Disk 1 that is 2.00 TB in size showed up in the StableBit Scanner. Wasn't exactly sure what this is or if it will just go away on its own at some point.

...

On a perfectly duplicated pool, the pass just enumerates all the files to make sure they're duplicated correctly. It should be pretty fast.

The Scanner sees the "virtual" physical disk. I realize that what I just said is a contradiction. 2.0 TB is the fixed size of the emulated physical disk that the pool is on. Since there really is no physical disk, this size is of no consequence.

The Scanner will be updated eventually to recognize this disk as a DrivePool disk.

Alex

Doug said:
Edit 1- almost 2hrs still indexing- Remote Web Access Media Library numbers are changing with indexing.

Indexing is a "Windows Search" process. It is affected by how "busy" it thinks your computer is. DrivePool has no control over the indexing speed.

WHS always indexes everything that's shared, DrivePool just shows the status.

Indexing is not required for the pool to function. DrivePool shows you a pie chart based on the Windows Search index when you select a folder.

Alex

phrozen087 said:

ajuhawk said:
Now that my system has sat for a few hours everything seems to have cleared up except for the fact that 687 bytes of data say there are "Unduplicated files" on one of my hard discs. The drives all say healthy though (I would think they would say unhealthy if something wasn't duplicated) so I am not sure what this is, maybe it will clear up eventually also on its own.

Windows tends to put some metadata on the disk, in a non-shared folder. Anything placed on the pool in a folder that's not duplicated is considered un-duplicated data. This is probably what you're seeing.

The recycling bin is also un-duplicated, if that's enabled.

Alex

IMPORTANT

Thank you all for testing early.

I've pulled the link for now because there's an issue with file renames. It's a very specific issue where if your pool is balanced equally across all drives, and something tries to rename a file while asking to overwrite the destination, there is a chance that the destination will not be overwritten properly.

Again, this is pretty specific, but the WHS backup engine renames files often so this needs to be fixed in order to maintain backup integrity.

I'll have this fixed up ASAP (1 to 2 days I think) and put up a new build.

Paaland

Just managed to get it then :) Migration went fine btw. And it's now indexing all the shares.

Alex

Saltydh said:
Not sure if I should have waited until the duplication processing was complete before resolving the warning. Either way I'm curious to know if there's any way of querying Drive Pool to ascertain the location of the duplicate files so I can manually remove them?? :D

Perform a "re-measure" from the DrivePool tab. If the duplicated usage doesn't go away after that finishes, then you must have a duplicated folder somewhere. (If you're sure that you don't then it's a bug)

DrivePool doesn't know exactly what files are duplicated. For efficiency, it doesn't keep that sort of fine grained statistics (that would require a database).

CoveFS monitors your pool for duplication issues in real-time and if it detects a problem, it notifies the DrivePool service which performs a full duplication pass over the entire pool.

A duplication pass will clean up files that are duplicated with too many file parts, so that's why I said that if you see space used by duplication, something must actually be in a duplicated folder somewhere.

Sg112

I migrate to M4 and update it went very well. Then I got a home server back up error a few hours latter; all back ups have errors. I tryed to repair but this does not work so I delete all back ups. Then I have 2 harddrives fail in about 2 hours. I put these into another server, but they are still the same both drive cause windows to stop responding for a long time. Can it be possible M4 cause damage to harddrives?

Saltydh

Thanks Alex,

I tracked down the duplicates after the re-measure produced the same result. No bugs to report here, everything came good after the duplicates were manually removed. :)

Cheers!

letho

Hey, i have not yet set up M3 on my whs, so i cannot migrate ;-) Can i set up m4 for testing now, or should i wait until the current issues are solved?

And one question, can i rename the drive letter?

Edit:

Ok, i have installed it :-) it seems to work and i have already renamed the pool drive letter. now i'm zrying to add the disk to the pool.

phrozen087

Saltydh said:
Thanks Alex,

I tracked down the duplicates after the re-measure produced the same result. No bugs to report here, everything came good after the duplicates were manually removed. :)

Cheers!

How did you find the duplicates manually? I was wondering if I could do something similar to find whatever this one tiny unduplicated file is on my pool (its this annoying 697 byte file, but it annoys me!)

In other news, after letting everything set over night I came back this morning and everything appears to be running well. Read/Write rates to the pool are also much faster then before, which is always a good thing.

I also found one additional minor bug: If the discs are asleep (I have my discs auto spin down after 10 mins if I am not using them) and I load the dashboard up fresh then Drivepool says I have no pool or discs in the pool. If I go access a disc or share, then the pool wakes up and if I reload the dashboard everything is normal. Doesn't really affect anything, but appears to be a bug. Also for reference, not sure if this has anything to do with it, my pool is 8 discs connected via eSata with 4 discs each inside 2 MediaSonic 4 bay enclosures. This behavior didn't occur in M3 ever.

Saltydh

phrozen087 said:

Saltydh said:
Phrozen087: How did you find the duplicates manually?

It's a corporate dedupe product I ran across the drives (in house stuff). Free alternative, NoDupe is something I've used before in a pinch. (MP3s and the like). http://www.otissoft.co.uk/products/index.html

Direct download link because Brothersoft is horrible:http://otissoft.co.uk/products/nodupe/bin/NoDupe_1_16_7_2.exe

Hope it helps.

dvdluvver

Just curious - are the bugs Alex is currently working out (that caused him to remove the M4 link) strictly migration related? I only ask because I have been sitting on the sidelines awaiting M4 to take the plunge with my WHS! Can't wait to get this going! My old WHSv1 (which I am using as my "duplication" system is starting to get pretty cranky.

Shane

The one Alex mentioned as the reason for him pulling the download link was not migration related. http://forum.covecube.com/discussion/comment/1517#Comment_1517

JazJon

dvdluvver said:
Just curious - are the bugs Alex is currently working out (that caused him to remove the M4 link) strictly migration related? I only ask because I have been sitting on the sidelines awaiting M4 to take the plunge with my WHS! Can't wait to get this going! My old WHSv1 (which I am using as my "duplication" system is starting to get pretty cranky.

Alex said:
"I've pulled the link for now because there's an issue with file renames. It's a very specific issue where if your pool is balanced equally across all drives, and something tries to rename a file while asking to overwrite the destination, there is a chance that the destination will not be overwritten properly.
Again, this is pretty specific, but the WHS backup engine renames files often so this needs to be fixed in order to maintain backup integrity.
I'll have this fixed up ASAP (1 to 2 days I think) and put up a new build."

I'm finally ready to update M3 to M4 as soon as the fixed version comes out :) The repair of M3 unduplicated files took me 2 days to complete.

Alex

Just an update.

The link was disabled because of a rename bug. This is not migration related.

I've fixed a number of issues in the latest build. Some were causing blue screens, and the rename bug.

It's being tested now, once it passes I will put it up.

Alex

Sg112 said:
...
Can it be possible M4 cause damage to harddrives?

DrivePool cannot physically damage hard drives.

It doesn't access drives directly, it accesses the files on the drives using file paths, just like other applications.

Symo

Have now updated from M3 to M4 no issues or worries so far, and it's not causing the issues that M3 was for me in crashing different services etc.

2.87 TB (or data)
5 Discs
10 Shares
336,000 Files
37,000 Folders

Migrate: 3mins
Measuring: 2hrs
Duplicating: 1hr
Indexing: 15hrs and still going

I love the fact that the pool is now a drive, it always annoyed me a bit it being a few sub-folders in on C: (and it looks much much nicer to have the pool looking like a plan old HDD). So the migration went fine, all shares appeared as they should and are available without issue...

Have been having issues with backups (before M4), not sure what was the cause of that, but once the newer M4 comes along I'll be trying to sort that out...

Thanks for all the hard work Alex, I am again very happy and feeling WHS2011 is now what it should have been!

ETA: Oh and a quick question for everyone:
Is anyone using 3TB drives (either now with M4 or previously with M3)?

I've got a 3TB drive that I added to my WHS just before I upgraded from v1 to 2011. v1 did not like the drive at all, even not in the pool it didn't like it...2011 picks it up as it should (2.73TB) but I haven't been game to add it to the pool so far, but I'm thinking I will now that it's been playing nicely and I don't see why Dp would have a problem with it. Thoughts?

Saltydh

Note: I'm running Storage Server.

2x 3TB drives here, no problems to report although these drives were migrated into the M4 pool rather than added as new drives.

Alex

Download is back with build 5387. Look at the top of this thread.

How to upgrade M4 to M4?

Install the new build.
Reboot the server.

I will update the wiki tomorrow and post a full change log. Fixed some crashes and the rename bug.

There is a long thread on this forum regarding boxee issues. I will look into this soon.

phrozen087

Alex said:
Download is back with build 5387. Look at the top of this thread.

How to upgrade M4 to M4?

Install the new build.
Reboot the server.

I will update the wiki tomorrow and post a full change log. Fixed some crashes and the rename bug.

There is a long thread on this forum regarding boxee issues. I will look into this soon.

Thanks for the update. Just finished installing it and rebooted. Was upgrading from the initial M4 release. The dashboard still says indexing, but I did a scan using SmartSync Pro 3 and it verified that all the data on the drives correctly matched the master copies I have been keeping while trying out the betas, so everything looks good for me. Still have this random 689 bytes of unduplicated data that I can't find, but hopefully I will eventually track it down. Thanks for your hard work, this product is awesome!

JazJon

Uhg, my M3 got unhealthy/un-duplicated files again today, it's been chugging away for hours again with the repair.

Is it possible to upgrade from M3 to M4 with unhealthy drives?

(or would it be faster to change shares from duplicated to not duplicated temporarily?)

I'm really excited to get my M4 going on with indexing, but I'm still stuck with unhealthy M3 drives. :(

hooj

I'm getting the following error when upgrading from the latest M3 to M4.

The add-in cannot be upgraded.

I have tried removing the M3 and reinstalling M4, but no drives were listed as available. I then removed M4 and reinstalled M3 and attempted the upgrade again, but encountered the same error :-(

Symo

Hmm...the new M4 is causing the same issues as M3 was in the end for me... (account logging out (I'm guessing from the different services crashing), dashboard doesn't want to start at all), when I first installed it it said a drive that I had removed before I installed it was "removing" so maybe that is the root of the issue. Have uninstalled the second M4 and will reinstall the first M4 and see what it says...

Alex one small note if it's of any concern, since dashboard wouldn't start I went to programs and features to uninstall it and the version was showing as 1.0.5387 ie missing one .0, not sure if that matters to you or not...

JazJon

Success, I'm now on M4 and old M3 shares migrated successfully. Indexing is happening, and I'm able to add library's into my Win 7 laptop at last.

I had one unexpected server reboot, and submitted a ticket with full logs. I'll leave that to Alex in the one on one support ticket. Things have been fine after the one reboot so far. I'll let things fly over night and see how it goes from there.

bergerac56

I just made the migration from M3 to M4 with the lastest build. It went like a charm. As the pool is now seen as a disk, I wanted to give it a name. This does not seem to work. Is it normal ?

ajuhawk

I also have not been able to label the pool drive - would be nice if it would have a label by default (DrivePool) to make it stand out. I also manually moved the drive letter to keep it away from hard drive additions/subtractions - Alex, you might think about using the last available drive letter by default instead of the first... also could it be user select-able (advanced install option?) during setup? or an option somewhere to change it? The manual method works, but...

Philmatic

I upgraded from M3 to the build of M4 built today, everything seems to work fine except the only folder that got moved to the D: drive was the "Client Computer Backups" share. All the other shares are still pointing to C:\ServerPool\ServerFolders\xxx and if I browse to that folder, it's still clearly a junction point to the old M3 style folder:

C:\ServerPool\ServerFolders\Movies --> \??\C:\ServerPool\ServerFolders.Mount\Movies.1

Should I reboot?

lee1978

Hi,

Just installed the new M4 i only had 1 problem and that was with the client backups but i think i have that sorted now otherwise all is running 99% ok tho there are only 2 things i see as misssing parts

1. The new Drive can not be renamed

2. The pool tab no longer shows the names of the Drives installed on the server this was very useful if there was a problem with a drive you knew which 1 it was and could remove it Alex could you please add this back on

Great work Alex

Lee

ajuhawk

Philmatic said:
I upgraded from M3 to the build of M4 built today, everything seems to work fine except the only folder that got moved to the D: drive was the "Client Computer Backups" share. All the other shares are still pointing to C:\ServerPool\ServerFolders\xxx and if I browse to that folder, it's still clearly a junction point to the old M3 style folder:

C:\ServerPool\ServerFolders\Movies --> \??\C:\ServerPool\ServerFolders.Mount\Movies.1

Should I reboot?

Philmatic - I had this same issue initially. Here's what Alex had me do (and it works):

You can simply right click on the folders in the pool root, go to properties and share them.

Click Advanced Sharing -> Click "Share this folder".

Before clicking ok, click "Permissions" and make sure allow "Full Control" is checked.

You should do this for each folder that was not correctly re-shared.

lee1978

Seem to be haveing problems with the client backups all was fine on M3 now none of my computers will backup if the client backup folder is on the pool plus for some odd reason the folder gets this added to it when i move it or recreate it to the pool

Client Computer Backups.DELETE.{F3D68A23-FA5D-4A66-95B8-85AF7868CE70}

the folder is locked when on the pool preventing any changes to it

Lee

bergerac56

Playing a bit more with M4, I got the following when moving a folder: The folder was moved successfully with some errors / Cannot enable shadow copy on the hard drive.

Then when I re-move the folder from the pool to another disk, there is no problem at all. This happens only when I move a folder to the pool. (tried a lot of times)

When I look at the drive representing the pool, I see the folder existing at time of M3 (I had only one very large folder in the pool) and a folder named "ServerFolders" containing the new moved folder and all other folders I could move.

Why this difference between a pre-M4 pooled folder and M4 pooled folder ?

Moreover, I noticed a temporary strange behavior. The pre-M4 folder was not referenced as pooled for a moment. After some closing and restarting of the dashboard and changing the views between "Integrated user interface" where the pre-M4 folder was marked Pooled and the other view where it was not included in the pooled folders, eveything was OK. Does this mean something?

JazJon

I just sent in the following support ticket, but wanted to share here with everyone as well.

_________________________

two problems (and 3rd question)

1) Can't repair Client Backups, or backup computer any more.

Errors exist in Client Computer Backup

Computers and Backups (JazPad is critical)

Additonal client computer backups tasks

Tools, Repair now,

Repair the Backup Database

The wizard did not complete successfully

Backup repair did not succeed

The wizard cannot repair the errors in your computer backups

Can you help me delete/reset the backup for JazPad? I want to have you double check things got migrated and or are in the correct location.

2) Even though new default shares are in libraries on media center, I see no shares via media center extenders (xbox 360)

Can you help me figure out how to force WHS Media Player library/database/index to reset itself again? You helped me fix this awhile back but maybe we can get it right now in M4

3) Can I safely rename the default pool drive letter from M: to P: ?

lee1978

@ jazon

if you move the client backup folder off the pool it works fine and you can repair it without problems seems the pool does like the client backup folder and it was fine on M3 easy fix for alex one thing i did notice was the backup gets to around 30% for me then a get and interuption error what ever that means hope that helps.

Lee

Philmatic

I'm also having a problem with client backups...

Howdy, Stranger!

Categories

Our Sites