Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In with Google

Poll

No poll attached to this discussion.

In order to better support our growing community we've set up a new more powerful forum.


The new forum is at: http://community.covecube.com


The new forum is running IP.Board and will be our primary forum from now on.


This forum is being retired, but will remain online indefinitely in order to preserve its contents. This forum is now read only.


Thank you,

Status of BETA M3

24

Comments

  • Yes Shane, seems if I had a new file to the drives, it makes it unhealthy, and then there is certain new files to cause it to go unhealthy over and over again.
  • edited July 2011 Resident Guru
    To clarify, if you create - right now - a new pooled, duplicate-enabled share, then create a text file in it, and then modify that text file, does the drive become unhealthy? And if (after repairing) you repeat with a non-duplicate-enabled share, does the drive become unhealthy?

    Hmm. While my personal suspicion is NTFS ACL issues from DE, I could be completely off-base, so it might be best if both of you could submit a log file to Alex and he can get to the heart of the problem. http://wiki.covecube.com/StableBit_DrivePool_Log_Collection
  • Covecube
    Status update on M3 as of today (7/11/11):

    Everything mentioned in the last update is coded. There's going to be one more thing that will be added and that's data visualization. It's already in progress and will be completed today or tomorrow.

    Once that's done, everything will need to be tested and any issues will need to be fixed up. Then it goes public.
  • Alex, Can you explain the main workflow of DrivePool. 

    I just logged in to my server and noticed excessive CPU consumption by DrivePool service. In ResourceMonitor I also noticed that it was accessing the pool contents. However those files were not modified after pool creation. Please explain what was the service actually doing.

    Thank you!
  • edited July 2011 Member
    Hi Alex,

    Long time WHS V1 user and reseller, have to say I think your product looks like a winner to me.
    I am eager as is everyone else seemingly on your forum to hear you say it is stable and ready for main stream use.

    Curious as to how big of a pool has been attempted with your product at this point, and across how many drives.

    Best Regards,

    Stacy Laboucane
    Internet Guys Inc.
  • Covecube
  • edited July 2011 Covecube
    ...
    I just logged in to my server and noticed excessive CPU consumption by DrivePool service. In ResourceMonitor I also noticed that it was accessing the pool contents. However those files were not modified after pool creation. Please explain what was the service actually doing.

    Thank you!
    The job of the DrivePool service is to access the pool :)

    In M2 the service does nothing except service I/O requests to the pool. Nothing else.

    CPU usage is caused by something accessing pooled files / directories.

    Of course if you suspect something going wrong then report it using http://stablebit.com/contact

    M4 will aim to bypass the service for CPU consuming I/O tasks, thus speeding everything up.

    ---

    M3 is more complicated and that's a different discussion. I will post more info when it's out.
  • Covecube
    Hi Alex,

    Long time WHS V1 user and reseller, have to say I think your product looks like a winner to me.
    I am eager as is everyone else seemingly on your forum to hear you say it is stable and ready for main stream use.

    Curious as to how big of a pool has been attempted with your product at this point, and across how many drives.

    Best Regards,

    Stacy Laboucane
    Internet Guys Inc.
    Stacy,

    Thanks for your support.

    Everything is moving as quickly as possible while trying not to compromise the design :)

    As far as pool size, I'm sure some of the beta testers on here can claim the pool size crown. Internally DrivePool testing is concentrating on interface diversity not total size. SATA / SSD / USB 2.0 / 3.0 are used for testing. The pool size is only a few terabytes.

    I have to say, I'm pretty excited about M3, once it's stabilized I think it will have all the core features that people have been asking for.

    I think we can look forward to testing with 16 TB+ once it's ready.
  • Alex, I understand that service was accessing the pool which introduces CPU consumption, that's totally normal, but I'm curious why does it access those files. I'm 100% positive that nothing I'm my network was accessing any on shares on the server.

    Thanks, Ruben

    Also, when is M3 scheduled? It looks awesome :)
  • M3 does look pretty good, and I'm looking forward to it.
  • Member
    M3 looks very good! Great job!
  • Member

    Stacy,

    Thanks for your support.

    Everything is moving as quickly as possible while trying not to compromise the design :)

    As far as pool size, I'm sure some of the beta testers on here can claim the pool size crown. Internally DrivePool testing is concentrating on interface diversity not total size. SATA / SSD / USB 2.0 / 3.0 are used for testing. The pool size is only a few terabytes.

    I have to say, I'm pretty excited about M3, once it's stabilized I think it will have all the core features that people have been asking for.

    I think we can look forward to testing with 16 TB+ once it's ready.





    Hi Alex,

    Thanks for taking the time and making the effort to address such a screw up as MS dropping the DE. In reality it looks like your product will be superior so that is awesome and you turned it into a business opperunity too boot ..Kudos to you :)

    I currently run 2 WHS V1 machines with 17TB, (one is a synctoy back up of the first) and have several clients with similar setups. One drawback has always been that V1 did not support the new SATA6 drives, so we have had to attach the faster drives to slower SATA ports for them to work.

    So when they annouced WHS v2.0 I was very excited to see it was on 64bit and would see the large, faster drives natively. Then they dropped DE ! Ack ! What a rollercoaster ride...

    At any rate seems we are on an up swing now thanks to you and your product.

    Keep up the good work !

    Stacy Laboucane
    Internet Guys Inc.




  • edited July 2011 Covecube
    Great news. M3 is now being finalized and should be released within days.

    Here are some of the new features:

    • DrivePool folders are now recognized by WHS as regular folders. This means that DrivePool folders will work with media streaming and remote web access just like any other folder.
    • Enable / disable pooled folder duplication without recreating the folder.
    • Migrate any folder to and from the pool, including the default WHS folders.
    • Data visualization. See folder usage, drive usage, and pool usage visually using pie charts.
    • File balancing. Efficient file balancing in order to ensure maximum usage of all available disk space.

    There's really a lot more to say here, especially about file balancing. I'll post more info once M3 is out.
  • Covecube
    Alex, I understand that service was accessing the pool which introduces CPU consumption, that's totally normal, but I'm curious why does it access those files. I'm 100% positive that nothing I'm my network was accessing any on shares on the server.

    Thanks, Ruben

    Also, when is M3 scheduled? It looks awesome :)
    The M2 service should not access anything on its own. It's essentially acting like a proxy for the pooled drive. You might want to try using Process Monitor to see exactly what files are being accessed.

    One exception is when you perform tasks. Like removing a drive from the pool or integrating a foreign disk. In those cases while service is performing the task asynchronously it will be accessing files on the pool. But tasks will terminate once they're complete or there's a problem encountered.

    Once a task is started, it does not require user interaction and will continue to run regardless of whether the Dashboard is active or not.
  • Covecube
    Hi Alex,

    Thanks for taking the time and making the effort to address such a screw up as MS dropping the DE. In reality it looks like your product will be superior so that is awesome and you turned it into a business opperunity too boot ..Kudos to you :)

    I currently run 2 WHS V1 machines with 17TB, (one is a synctoy back up of the first) and have several clients with similar setups. One drawback has always been that V1 did not support the new SATA6 drives, so we have had to attach the faster drives to slower SATA ports for them to work.

    So when they annouced WHS v2.0 I was very excited to see it was on 64bit and would see the large, faster drives natively. Then they dropped DE ! Ack ! What a rollercoaster ride...

    At any rate seems we are on an up swing now thanks to you and your product.

    Keep up the good work !

    Stacy Laboucane
    Internet Guys Inc.




    Thank you. I hope the final version will live up to everyone's expectations.

    The current BETA M2, while functional, visually looks a bit incomplete. BETA M3 should fill in the gaps in terms of functionality and missing UI.

    I can't wait to get BETA M4 out so that the original vision for DrivePool is complete! It has been a challenge making DrivePool a reality, but I hope that some time in August we will have a working BETA M4. After that it's release time.
  • Covecube
    Ok, I had to post this:

    image

    Working out the kinks :)
  • edited July 2011 Member

    Well this is drivebender but I am checking out both.

    Bet I have more drive space than you!

     

    https://picasaweb.google.com/102122954229684497539/PBs?authuser=0&authkey=Gv1sRgCNyFjbHcuNvpgAE&feat=directlink

     

  • Alex, I understand that service was accessing the pool which introduces CPU consumption, that's totally normal, but I'm curious why does it access those files. I'm 100% positive that nothing I'm my network was accessing any on shares on the server.

    Thanks, Ruben

    Also, when is M3 scheduled? It looks awesome :)
    The M2 service should not access anything on its own. It's essentially acting like a proxy for the pooled drive. You might want to try using Process Monitor to see exactly what files are being accessed.

    One exception is when you perform tasks. Like removing a drive from the pool or integrating a foreign disk. In those cases while service is performing the task asynchronously it will be accessing files on the pool. But tasks will terminate once they're complete or there's a problem encountered.

    Once a task is started, it does not require user interaction and will continue to run regardless of whether the Dashboard is active or not.

    Yeah, actually that was my initial question. I'd like to get more familiar about those tasks that DrivePool performs. Are you planning to document it and have it available to public? 

    P.S. Is that a screenshot from youtube or google datacenter using DrivePool? :) :) :)
  • Covecube
    Yeah, actually that was my initial question. I'd like to get more familiar about those tasks that DrivePool performs. Are you planning to document it and have it available to public? 

    P.S. Is that a screenshot from youtube or google datacenter using DrivePool? :) :) :)
    If I understand your original question correctly, you are asking what tasks DrivePool performs automatically in the background?

    The answer to that is none, in BETA M2.

    In M2 all the tasks are started from the Dashboard GUI and are documented in the various blog posts (especially the last 2).


    To sum it up, every wizard is a service task. DrivePool does as little as possible in the UI and hands off all work to the background service. Keep in mind, these are manually triggered by the user.

    The most obvious one is the Remove drive wizard.
  • edited July 2011 Covecube
    P.S. Is that a screenshot from youtube or google datacenter using DrivePool? :) :) :)
    Yes, Google called us and they want to make a backup of the Internet... Just in case :)
  • Alex, I got another question. Lets say I shutdown/restart the system. How does DrivePool determine healthy status of my pool? I mean by the time DrivePool was turned off, some of duplicated files could get deleted/damaged?
    Will DrivePool try to rescan all files under pool and mark those files as unhealthy even if user don't access the pool at all?
  • Resident Guru
    Not Alex, but if it's a normal (i.e. requested) shutdown/restart, Windows says "hey, all processes, shutting down, please finish what you're doing and let me know", and DrivePool says, "okay... alright, finished up here, it's safe to turn me off now".

    If on the other hand it's a BSOD or you've tripped over the power cord, it'll depend on DrivePool's ability to keep track of its state. "Hmm, okay, user wants me to copy file A to folder B, making a note... okay, finished that, making a note... okay, another file to copy, making a note... *Argh! BSOD*... (system reboots, DrivePool starts up) okay looks like I didn't shutdown properly, let me check my notes..."
  • edited July 2011 Resident Guru
    @wordgasm - holy batdrives batman... 35.9 PB?! That's some serious hardware! For real? :)

    (what's even scarier is the "2.80 TB free" in context of the above)
  • edited July 2011 Covecube
    Alex, I got another question. Lets say I shutdown/restart the system. How does DrivePool determine healthy status of my pool? I mean by the time DrivePool was turned off, some of duplicated files could get deleted/damaged?
    Will DrivePool try to rescan all files under pool and mark those files as unhealthy even if user don't 
    access the pool at all?
    DrivePool does not actively scan the entire pool for consistency.

    Every time you open a duplicated file DrivePool performs a quick sanity check on the file to make sure that all the duplicated parts are consistent. If the file is not consistent then it's tagged with some external metadata (not on the file, in C:\ProgramData\...) marking the drive and the particular file suspect.

    So if you were to shut down, damage some duplicated files and restart, DrivePool would detect the problem at the time that the damaged duplicated file would be accessed. Since it's a duplicated file the access would succeed, but the file would be tagged as damaged. It would be re-duplicated with the good part after running the repair wizard.

    This is similar to what would happen on a non-pooled volume if a sector went bad while the system was turned off. You wouldn't know it until you accessed the file.

    ---

    Just to comment on what Shane added. Yes, DrivePool does have a safe shut-down sequence that unmounts the pool safely and shuts down all the internal "Services". The main service is modular and it contains a suite of mini-service-like modules each interacting with each other, so that it's easy to extend the code and add new functionality.

    In the event of a power cord pull, the worst case scenario that I can think of is that you were in the middle of writing a duplicated file. What would happen is the NTFS metadata could get corrupted, but because NTFS is a journaling file system, it would repair itself. NTFS guarantees the non-corruption of its metadata (not the files on it). DrivePool would have a corrupt duplicated file, but it wouldn't matter much because any file that was being written as the system powered off is most likely unusable. It would be tagged corrupt as described above, but you would almost certainly just delete it and start over.
  • Alex, do you think you expose a feature to have an ability to scan entire pool/folder/file for consistency upon user request? This might be a good feature to keep your data safe. Otherwise your hdd can go bad, and your data can get corrupted without you even knowing it (if you don't access corrupted files).
  • Hi, Alex - it's been awhile!

    I have two questions:

    1 - M3 is close, right?  :)
    2 - when removing a drive from the pool, do the shares remain available?   I remember a big pain with WHS v1 was that if you wanted to remove a drive, the whole server was essentially taken offline (which was not appreciated by my family!)

    I have my old WHS v1 serving as a backup of all my data (some 8TB worth), so I'm going to try M3 in my "live" WHS 2011 server (with some duplicated folders and some not, knowing that if everything crashes I can retrieve my backups) to see how she flies.   Looks like it's really coming together nicely (and my suspicion is that this is going to be the most desired option for DE replacement for the masses!)! 
  • Covecube
    Alex, do you think you expose a feature to have an ability to scan entire pool/folder/file for consistency upon user request? This might be a good feature to keep your data safe. Otherwise your hdd can go bad, and your data can get corrupted without you even knowing it (if you don't access corrupted files).
    That's a big question, perhaps we should start a new discussion on this point.
  • edited July 2011 Covecube
    Hi, Alex - it's been awhile!

    I have two questions:

    1 - M3 is close, right?  :)
    2 - when removing a drive from the pool, do the shares remain available?   I remember a big pain with WHS v1 was that if you wanted to remove a drive, the whole server was essentially taken offline (which was not appreciated by my family!)

    I have my old WHS v1 serving as a backup of all my data (some 8TB worth), so I'm going to try M3 in my "live" WHS 2011 server (with some duplicated folders and some not, knowing that if everything crashes I can retrieve my backups) to see how she flies.   Looks like it's really coming together nicely (and my suspicion is that this is going to be the most desired option for DE replacement for the masses!)! 
    1. The binary is ready to go, it's 2AM here, so Tomorrow :)
    2. When removing a drive from the pool the entire pool is not taken offline. All the unduplicated files on the drive being removed become inaccessible until the removal process is complete. Duplicated files remain accessible but may not have duplication until removal is complete.

    As a followup to 2, now that M3 is complete, I should add that when balancing folders the system moves files around. As a result it will lock an individual file for the duration of the file move and then unlock it. If a file cannot be locked because something has it open then it's not moved in the balancing pass.

    Balancing is only performed when necessary. Necessity is defined by a formula based on data coming from the pool in real-time. In other words, no balancing every 30 minutes. Automatic balancing is limited to every 24 hours, when needed, and only at night. Manual balancing can be started at any time from the UI.
  • While we wait for M3 to be posted.........Alex can you give us an idea on cost to purchase the Ad-on?
  • I recall an earlier figure of $20 in the early development releases.

     

Sign In or Register to comment.