Log In     Register    

DarkMX Support Forum
Questions and comments about the software
<<  Back To Forum

more robust dat file handling, or get away from dats all togethe

by Guest on 2021/03/04 03:51:02 AM    
an annoying issue when pcs crash is open programs sometimes loose there currently open documents.
DarkMX, Fopnu. Tixati, Winmx all suffer from this fate when the PC unexpectedly restarts or even sometimes a normal restart without closing the programs first upon next start the user finds themselves having to re setup the program from scratch.
Its a minor inconvenience but a real bummer. is there a method to prevent this? maybe a backup dats that is created daily so if the primary dats fails to load the backup is used instead of starting fresh.

Just a suggestion.
by Guest on 2021/03/11 04:24:09 AM    
One peculiarity I noticed with Windows and NTFS during crashes: the file size tracked by the FS is changed, but no data is actually written.

The example is based on a text file:
*many lines with data
*one last line of eg. 12 characters added (+ 2 bytes for CRLF)
*this file now has 14 NULL bytes at the end of it

Maybe this is what ultimately breaks Tixati and others and with only little detection code you could fall back to the "last successful DAT files"
by KH on 2021/04/17 04:56:48 PM    
It's odd that this would happen on your system.

We don't actually just save the .dat file directly.

We save to .dat.temp, and then once that's fully written and closed, we rename the file to .dat, overwriting the old .dat in an atomic way.  This applies for both Linux and Windows builds.

When you start the program successfully, it always backs up the .dat files too.  Look in the config folder and you will see .lastloadok.dat which is just a direct atomic copy made upon startup.

Also, upon restart, if it reads a corrupt .dat file, it should easily detect it (there's a check-hash in the file header) and present a dialog window that asks if you want to load the last successfully loaded file.  You can test this out by using a hexedit and altering a few bytes of a .dat file and see what happens upon program start.

The only time you'll have to start over is if you run the program the first time, get everything setup, and then the computer crashes before the very first time a .dat file is saved in the background.  But that's only a dozen or so minutes for a full background save cycle to complete.  So at any given time, you shouldn't lose more than the last 10-15 minutes of changes IF you computer were to crash or lose power.

There's also in Help > Diagnostics a file operations log, so you can see how/when it saves the dat files.  If there's anything funny going on in there, I'd like to hear about it for sure.
by Guest on 2021/11/02 02:31:47 AM    
This part of the ext4 documentation is maybe relevant:
auto_da_alloc(*), noauto_da_alloc

   Many broken applications don’t use fsync() when replacing existing files via patterns such as fd = open(“foo.new”)/write(fd,..)/close(fd)/ rename(“foo.new”, “foo”), or worse yet, fd = open(“foo”, O_TRUNC)/write(fd,..)/close(fd). If auto_da_alloc is enabled, ext4 will detect the replace-via-rename and replace-via-truncate patterns ...
If you follow that note to manual for POSIX close(2):
A successful close does not guarantee that the data has been successfully saved to disk, as the kernel uses the buffer cache to defer writes. Typically, filesystems do not flush buffers when a file is closed. If you need to be sure that the data is physically stored on the underlying disk, use fsync(2). (It will depend on the disk hardware at this point.)
Do your programs follow this procedure?
by stumpy on 2021/11/06 03:30:23 AM    
On almost all Linux filesystems, ext3, ext4 (with default options), btrfs, etc, the rename is guaranteed to be committed to disk only after the file blocks are written.  You will be guaranteed after a crash to either have the old file, or the new file.

The use of fsync at one time was a bit of a stop-gap measure to force correct ordering of the write operations (as a side-effect), but it's not needed these days, and can be a problem in some rare cases.  There's a big performance hit on some file systems (eg. ext3) because it will cause a flush of all unrelated file blocks too, which could take several seconds, holding your process hostage.

If you're only concerned with making sure the file blocks are written before the rename is written, but not that it is done right away, the fsync is completely unnecessary.  Those old ext4 docs where it asserts that apps that don't use fsync are "broken" is quite incorrect, which is why it was later patched to guarantee the expected behavior.


fsync can block for a very long time:

Windows is another story.  One must use MoveFileEx with MOVEFILE_REPLACE_EXISTING, or there are some newer API calls if you don't need to run on XP.  Not sure if NTFS needs a sync on the temp file before the move or not, but probably not.
by KH on 2021/11/08 03:44:15 AM    
For the person who asked about fsync, yes, we do call that before the close/rename, when saving a config file when the program is closing or terminating on sys shutdown.  Same on Windows too, we call FlushFileBuffers.

However, it really isn't necessary at all and may get removed in the future.

And in some cases, it can cause a hell of a holdup in your process, even if you're using a worker thread to do the waiting.  We're talking 30+ seconds in some situation (eg. using a slow flash drive during a pre-existing large file copy operation.)

All the major filesystems will not commit to disk the rename before the data blocks of the temp file.  Ext4 was patched just after it came out 10+ years ago because it was originally broken in the way it sequenced the operations.  BSRFS, ext3/4, zfs, etc. are all good now.  When replacing, either the old file will still be there, or the new fully-written one will.

This web site powered by Super Simple Server