Incomplete File

Comments

• ethanol

  February 2012 Member

  Got a very helpful response from Alex which I expect he will clean-up and post to the blog : 

  
  

  
  I realize that the new M4 duplication process is a bit of a mystery. I plan to do a full write up on the blog once I get a moment.

  
  

  First, let's assume you're running with real-time duplication enabled (DrivePool settings). That's the default.

  
  

  In short, here's how it works:

  • On a brand new pool, when you copy a file into a duplicated folder, the file is written to 2 drives at the same time in parallel.
  • Whenever you delete / move or perform any I/O on a duplicated file, that I/O is performed, in parallel, over both files.
    
    This means that the 2 file parts for a single duplicated file are always in sync without any background duplication.
  • Whenever you change the duplication setting of an existing folder, DrivePool needs to go through each file in that folder and make sure that it's duplicated accordingly.
    
    This is the "Duplicating..." message that you see towards the bottom.
    
    Any new files placed into that folder get duplicated in real-time, as explained above as soon at you enable duplication, but we need to make sure that any existing files are duplicated appropriately.

  • DrivePool performs a quick sanity check whenever any file opened to make sure that it has the correct number of file parts. E.g. A duplicated file should have 2 file parts, a non-duplicated file should have 1 file part.
    
    If it finds a file with the incorrect number of file parts it sends a message to the DrivePool system service essentially saying incomplete file found (that's the warning that you see in the log).
    
    When DrivePool sees this message, if it's not already performing background duplication, it queues a run immediately.
    
    It is normal to see this in the log for folders that have just had their duplication level changed. That's because the background duplication run is opening each file to query it for duplication status and some of those are "incomplete". The duplication engine ignores this message in these cases because it expects it.

  What is considered consistent?

  • For all file parts to be considered consistent, they must have the same size and modification time.
  • If the modification time is not the same, the background duplication engine performs a full file hash of all the involved file parts to check whether the file contents differ.

  What can go wrong?

  • If a file is open exclusively, background duplication can't work on it so it skips it for this run. Remember, this is background duplication only and doesn't apply to normal pool usage.
    
    You will see this in the form of a warning at the end of the run.
  • If a file is inconsistent (as outlined above) you will get a "file conflict". This will show up as a list of file parts that don't match. You will be given the option for DrivePool to automatically resolve the conflict by deleting the older file part and restoring duplication using the newer one.
    
    Alternatively, you can examine the file parts manually by opening the PoolPart... folder in explorer and deleting the one that's incorrect.
    
    This is a very rare condition that should never happen under normal usage.
    
    This should only happen if something modifies the files on one of the pool parts and does not change the other copy of the same file, causing a mismatch.

   

  Share on Twitter