Incomplete File

ethanol

Running M4 5669

Looking in the Service logs I have pages and pages of the following :

I've submitted the logs and a bug report - anyone else encountered this?

DrivePool.Service.exe Warning 0 [CoveFsPool] Create found an incomplete file. 2012-02-05 09:05:51Z 880202838457

DrivePool.Service.exe Warning 0 [CoveFsPool] Create found an incomplete file. 2012-02-05 09:05:51Z 880202849223

DrivePool.Service.exe Warning 0 [CoveFsPool] Create found an incomplete file. 2012-02-05 09:05:51Z 880203190074

DrivePool.Service.exe Warning 0 [CoveFsPool] Create found an incomplete file. 2012-02-05 09:05:51Z 880203217605

DrivePool.Service.exe Warning 0 [CoveFsPool] Create found an incomplete file. 2012-02-05 09:05:51Z 880203234888

DrivePool.Service.exe Warning 0 [CoveFsPool] Create found an incomplete file. 2012-02-05 09:05:51Z 880203243512

DrivePool.Service.exe Warning 0 [CoveFsPool] Create found an incomplete file. 2012-02-05 09:05:51Z 880203259738

DrivePool.Service.exe Warning 0 [CoveFsPool] Create found an incomplete file. 2012-02-05 09:05:51Z 880203286772

DrivePool.Service.exe Warning 0 [CoveFsPool] Create found an incomplete file. 2012-02-05 09:05:51Z 880203312553

DrivePool.Service.exe Warning 0 [CoveFsPool] Create found an incomplete file. 2012-02-05 09:05:51Z 880203330991

DrivePool.Service.exe Warning 0 [CoveFsPool] Create found an incomplete file. 2012-02-05 09:05:51Z 880203340973

DrivePool.Service.exe Warning 0 [CoveFsPool] Create found an incomplete file. 2012-02-05 09:05:51Z 880203356534

ethanol

Got a very helpful response from Alex which I expect he will clean-up and post to the blog : 

I realize that the new M4 duplication process is a bit of a mystery. I plan to do a full write up on the blog once I get a moment.

First, let's assume you're running with real-time duplication enabled (DrivePool settings). That's the default.

In short, here's how it works:

• On a brand new pool, when you copy a file into a duplicated folder, the file is written to 2 drives at the same time in parallel.
• Whenever you delete / move or perform any I/O on a duplicated file, that I/O is performed, in parallel, over both files.
  
  This means that the 2 file parts for a single duplicated file are always in sync without any background duplication.
• Whenever you change the duplication setting of an existing folder, DrivePool needs to go through each file in that folder and make sure that it's duplicated accordingly.
  
  This is the "Duplicating..." message that you see towards the bottom.
  
  Any new files placed into that folder get duplicated in real-time, as explained above as soon at you enable duplication, but we need to make sure that any existing files are duplicated appropriately.

• DrivePool performs a quick sanity check whenever any file opened to make sure that it has the correct number of file parts. E.g. A duplicated file should have 2 file parts, a non-duplicated file should have 1 file part.
  
  If it finds a file with the incorrect number of file parts it sends a message to the DrivePool system service essentially saying incomplete file found (that's the warning that you see in the log).
  
  When DrivePool sees this message, if it's not already performing background duplication, it queues a run immediately.
  
  It is normal to see this in the log for folders that have just had their duplication level changed. That's because the background duplication run is opening each file to query it for duplication status and some of those are "incomplete". The duplication engine ignores this message in these cases because it expects it.

What is considered consistent?

• For all file parts to be considered consistent, they must have the same size and modification time.
• If the modification time is not the same, the background duplication engine performs a full file hash of all the involved file parts to check whether the file contents differ.

What can go wrong?

• If a file is open exclusively, background duplication can't work on it so it skips it for this run. Remember, this is background duplication only and doesn't apply to normal pool usage.
  
  You will see this in the form of a warning at the end of the run.
• If a file is inconsistent (as outlined above) you will get a "file conflict". This will show up as a list of file parts that don't match. You will be given the option for DrivePool to automatically resolve the conflict by deleting the older file part and restoring duplication using the newer one.
  
  Alternatively, you can examine the file parts manually by opening the PoolPart... folder in explorer and deleting the one that's incorrect.
  
  This is a very rare condition that should never happen under normal usage.
  
  This should only happen if something modifies the files on one of the pool parts and does not change the other copy of the same file, causing a mismatch.