Corrupt compressed files

[jklymak]

Sometimes the compressed files get corrupted for some reason. Currently if I do a big multi_dbd.get and one of these files is in there, it will get quite a ways through the translation and then just say "Unexpected reading error"

dbd = dbdreader.MultiDBD(pattern=f'./*.[d|e]cd',
                             cacheDir='.')
data = list(dbd.get_sync('sci_water_temp', 'sci_water_cond', 'sci_water_pressure'))

You can find a dataset that does this at https://cproof.uvic.ca/temp/bad.tar In this case, it is 02380108.* that is corrupt, and if you ignore that file, then dbdreader works fine.

it would be nice if these corrupted files were ignored (and maybe logged). Or at the very least the error message said what the problem file was.

I poked around some, and I couldn't quite parse when this was happening (sure in decompress.c), but it wasn't clear when the file was read and tripping up on this, and if it would be easy to ignore.

Thanks!

[jklymak]

Note a related issue was fixed at mousebrains/dbd2netcdf@429722f

[smerckel]

@jklymak , thanks for pointing this out. The problem itself is almost certainly not in dbdreader, but in the glider's science firmware. Tackling this might be like opening Pandora's box, because the software issues on the glider science can manifest themselves in many different places when trying to read the files. I am not sure what is the best way of handling this. I modified the C code such, that an error code is propagated back to the python caller, i.e. DBD._get() method calling _dbdreader.get().
At this stage, the options are:

• raise an Exception
• ignore the file
• ignore the file with a warning message

Personally I would opt for raising an Exception, to avoid processing data, thinking all is alright, where in fact some data may be missing. Perhaps in your case, you would not want that, in particularly when using MultiDBD, when you are left with finding out which files are misbehaving, and remove them accordingly.

[jklymak]

I'm fine with an exception and a message saying what file is bad. For the above there was one bad file out of hundreds (and it was in the middle)

It may take a few attempts if there are multiple bad files. But it's still easier than manually figuring out the bad file!

Thanks.

[smerckel]

Reading a compressed file that is corrupted (coded as a C extension) now propagates the error number back to the calling python method and raises an exception that allows further investigation. This behaviour may or may not be desired when using MultiDBD's get method. Default behaviour is that an exception is raised when a compressed file cannot be read. Using the flag "continue_on_reading_error" can be set to True to simply ignore the offending file. A logger.warning() will notifiy the user, however.

[jklymak]

OK, got to trying this in earnest, and now it has a different error, I'm afraid...

 File "/mounts/cproofdata/processing/deployments/dfo-marvin1003/dfo-marvin1003-20250925/process_deploymentRealtime.py", line 71, in <module>
    outname = slocum.binary_to_timeseries(binarydir, cacdir, l1tsdir,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cproof/pyglider/pyglider/slocum.py", line 920, in binary_to_timeseries
    dbd = dbdreader.MultiDBD(pattern=f'{indir}/{search}',
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cproof/dbdreader/dbdreader/dbdreader.py", line 1329, in __init__
    self._update_dbd_inventory(cacheDir, skip_initial_line)
  File "/home/cproof/dbdreader/dbdreader/dbdreader.py", line 1991, in _update_dbd_inventory
    dbd, result = self._safely_open_dbd_file(fn, cacheDir, skip_initial_lines, missing_cacheIDs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cproof/dbdreader/dbdreader/dbdreader.py", line 1949, in _safely_open_dbd_file
    dbd=DBD(fn, cacheDir, skip_initial_lines)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cproof/dbdreader/dbdreader/dbdreader.py", line 707, in __init__
    self.headerInfo,parameterInfo,self.cacheFound, self.cacheID = self._read_header(self.cacheDir)
                                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cproof/dbdreader/dbdreader/dbdreader.py", line 1071, in _read_header
    result = dbdheader.read_header(self.fp)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cproof/dbdreader/dbdreader/dbdreader.py", line 625, in read_header
    if not self.parse(fp.readline())=='dbd_label':
                      ^^^^^^^^^^^^^
  File "/home/cproof/dbdreader/dbdreader/decompress.py", line 225, in readline
    line = self.bytesIO.readline()
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cproof/dbdreader/dbdreader/decompress.py", line 190, in readline
    self.write(next(self.source))
               ^^^^^^^^^^^^^^^^^
  File "/home/cproof/dbdreader/dbdreader/decompress.py", line 81, in decompressed_blocks
    block = self._decompress_block(fp)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cproof/dbdreader/dbdreader/decompress.py", line 54, in _decompress_block
    b = lz4.block.decompress(fp.read(size), Decompressor.CHUNKSIZE)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_block.LZ4BlockError: Decompression failed: corrupt input or insufficient space in destination buffer. Error code: 5112

Note there is plenty of space, so it's failing due to a corrupt lz4 block.

[smerckel]

@jklymak, could you make an ebd file available that shows this behaviour? That would make it easier for me to find a fix for it, or at least handle the error properly. Thanks.

[jklymak]

Appologies - I should have done that in the first place:

https://cproof.uvic.ca/gliderdata/deployments/dfo-marvin1003/dfo-marvin1003-20250925/badcasts.tgz

• 02450127 and 02450137 are bad so far as I can tell.

Thanks!

[smerckel]

Upon trying to read the header of the file, the file gets decompressed first if needed. In some cases it seems that the decompression fails. Reading a header can fail in several ways and depending on the reason, an exception is raised or a warning is produced. Since in this case, it is likely that other files that are being called with MultiDBD will be read successfully, a decompression failure will only produce a logging.warning.

It appears that only 02450137.tcd is corrupt. Now it gets skipped completely, but there is still some useful information in there. I suspect that this has not so much to do with the compression algorithm itself, as with the still persistent problem in the glider firmware, at least up to 11.04, that from time to time (sometimes very often) data are not written to file. There is also a need to repair those corrupted files to extract as much data from them as possible, rather than just throwing all away. This change does not do that however.

[jklymak]

@smerckel This seems to work fine.

In general, I'm pretty happy if "most" of the data makes it through processing, since this is realtime data and just used to give an idea what is going on with at-sea gliders. Of course getting all the data from the post-deployment data is more important, and I don't think I've had too much problem with that.

Thanks again!

[zdenek-1983]

Hi there,

I am trying to tackle an issue that started in the same way as for @jklymak initially. I was trying to process *.scd and *.tcd files and started getting the "Unextected reading error" message. I did notice that there have been some updates in the repository since so I've pulled those and rebased my changes on top of that (I have a little interactive script using the dbdreader written in python). But now I am getting a different error. See below. This is now happening for the files that were processing previously without issue. Seems to be pointing to the py_get() method in dbdreader.c file. Would you have any thoughts on this?

Traceback (most recent call last):
File "/home/zedd/GliderProc/dbdreader/proctest_interactive.py", line 100, in
main()
File "/home/zedd/GliderProc/dbdreader/proctest_interactive.py", line 63, in main
data = dbd.get_sync(*required_params)
File "/home/zedd/GliderProc/dbdreader/dbdreader/dbdreader.py", line 1575, in get_sync
tv = self.get(*parameters, decimalLatLon=decimalLatLon, discardBadLatLon=discardBadLatLon,
File "/home/zedd/GliderProc/dbdreader/dbdreader/dbdreader.py", line 1451, in get
r_eng = self._worker("eng", *eng_variables, **kwds)
File "/home/zedd/GliderProc/dbdreader/dbdreader/dbdreader.py", line 2098, in _worker
t, v = i._get(*p, **kwds)
File "/home/zedd/GliderProc/dbdreader/dbdreader/dbdreader.py", line 977, in _get
error_no, r=_dbdreader.get(self.n_state_bytes,
ValueError: too many values to unpack (expected 2)

Thank you.

[smerckel]

Hi @zdenek-1983 ,

I must confess that I left things in an unfinished state regarding the handling of problematic compressed binary files. There are two issues. One is that until recently the C-code of dbdreader never caused an error (that is, the files thrown at it were always well behaved). Only during our last experiment with a new G3 glider, and compressed files and a microrider, we noticed that files got corrupt, triggering several different reading issues. I implemented the return of an error code so that the python end can handle these.
Done separately, is to try to get dbdreader back on track when something unexpected happens in a file. That has been proven a bit harder than anticipated, and constantly recompiling the C code does not speed up things. I got more urgent things on my desk and had to prioritise. Anyway, I will try to have a look at this later this week. It would be helpful if I could get access to the offending file.

Thanks

[zdenek-1983]

Hi Lucas, Thanks for getting back to me on the issue. I have to check with my colleague that I can provide the data as I am not the owner of it but will get back to you on that. Best regards, Zdenek

[smerckel]

Hi @zdenek-1983,
One offending data file would be sufficient. (I would also need the corresponding .cac file).
Thanks,
lucas

[zdenek-1983]

Hi Lucas,

Apologies, I was away for a bit last week. I'll send the files shortly. Not sure which one is the culprit but it will probably show up when you try to process it.

We did get around this issue when we removed the corrupted file as suggested elsewhere in the thread and would be quite OK to operate this way for the moment. However, as mentioned, there seems to be another issue with the "too many values to unpack" error that is a mystery to me. It shows on the realtime *.scd and *.tcd files (once the offending file is removed) as well as on the delayed *.ecd and *.dcd ones that were processing fine previously.

[zdenek-1983]

Hi Lucas,

files attached.

cache.zip

corrupted_files.zip

Thank you for looking into this.