On 01/14/2012 01:25 AM, Jef Driesen wrote:

On 01/13/2012 11:04 PM, Philip Balister wrote:

...

So I have the right cable now and I can download dives from my computer into subsurface.

Glad to hear it works now.

...

1. I have six dives in the computer, but only the first 5 show in

subsurface.

Depending on your (short) sample rate setting and (long) dive times, it's possible there is not enough memory to store profiles for all dives. In addition to the profile data, the oceanic dive computers maintain a summary for each dive stored in separate memory area. So it's possible the dive computer can show dives for which it doesn't have the profile data anymore.

Sample rate looks like 15 seconds. It is the most recent dive that does not show up.

I sent the dump files privately.

Philip

If you send me a memory dump of your device, I can have a look. Instructions are available here:

http://www.divesoftware.org/libdc/contribute.html#memory

...

2. There is no pressure information shown in subsurface.

The vtpro compatible devices don't record the tank pressure as far as I know.

Jef

It's this timestamp that is causing the trouble in your case. Normally with a

time based sample rate, there is a fixed number of samples per time unit. For example a 15s samplerate will always have 4 samples per minute. But that assumption is violated in your problematic dive. There are 5 samples with a timestamp of 37 minutes. I have absolutely no idea why the dive computer does this. It's the first time I encounter this.

I've seen this in some of the dives downloaded from my proplus2. Don't know what causes it. I've had it occur right at the beginning of a dive. i.e. time stamp = 1 minute.

On 01/15/2012 03:53 PM, Dirk Hohndel wrote:

On Sun, 15 Jan 2012 08:31:31 +0100, Jef Driesenjefdriesen@telenet.be wrote:

...

On 01/14/2012 07:57 PM, Philip Balister wrote:

...

On 01/14/2012 01:25 AM, Jef Driesen wrote:

...

On 01/13/2012 11:04 PM, Philip Balister wrote:

...

1. I have six dives in the computer, but only the first 5 show in

subsurface.

Depending on your (short) sample rate setting and (long) dive times, it's possible there is not enough memory to store profiles for all dives. In addition to the profile data, the oceanic dive computers maintain a summary for each dive stored in separate memory area. So it's possible the dive computer can show dives for which it doesn't have the profile data anymore.

Sample rate looks like 15 seconds. It is the most recent dive that does not show up.

I already found where it goes wrong. The dive is downloaded correctly, but the parsing fails, and I guess subsurface doesn't import anything at all in that case.

The problem is that the vtpro compatible dive computers support a depth based sample rate in addition to the time based sample rate. This means the dive computer doesn't record at fixed time intervals, but whenever a change in depth crosses a certain threshold. To be able to draw the profile graph, a timestamp is recorded for each sample. This "feature" is a real pain in the ass because it does break any application that assumes a fixed time interval between two samples. To make things even worse, the timestamp only has a resolution of one minute, which means there can be several samples with the same timestamp.

Yikes. Subsurface would be rather unhappy with this. We definitely assume that the time stamps are strictly monotone.

...

Another potential problem because applications are likely to expect the timestamps to be unique and increasing for each sample. To workaround this issue, the libdivecomputer code counts the number of samples within the same minute, and then spread them evenly.

So you give them an artificial 15 sec resolution?

I agree that the timestamps should be strictly montone. Having two or more samples at the exact same time doesn't make any sense. To avoid this problem with the depth based sample rate, the libdivecomputer code distribute those samples evenly within that minute.

The resulting (artificial) resolution varies depending on the number of samples that is present. For example if you have 3 samples with the same timestamp, they are spaced 60/3=20s apart. The next minute you could have 7 samples, and then they get spaced 60/7=8.57s (rounded to the nearest second because we don't support a higher resolution).

It gets worse if you stay at constant depth (or only very small changes within the depth threshold), because then you get no samples at all. This is problematic for drawing, because you typically draw a straight line between two samples, and thus the time without any samples will get drawn with a certain slope, rather than at a constant depth.

Last year, I wrote a detailed document describing all those sampling issues:

http://www.divesoftware.org/libdc/sampling.html

The missing samples would be very similar to surface intervals in section 2.3.4, where the only difference is that you are not at the surface but a certain depth of course :-)

...

It's this timestamp that is causing the trouble in your case. Normally with a time based sample rate, there is a fixed number of samples per time unit. For example a 15s samplerate will always have 4 samples per minute. But that assumption is violated in your problematic dive. There are 5 samples with a timestamp of 37 minutes. I have absolutely no idea why the dive computer does this. It's the first time I encounter this.

But that shouldn't be a problem, if you then spread those evenly and give us an artificial 12 sec resolution. Subsurface makes no assumptions that the sample rate has to be constant.

The spreading only happens for the depth based sample rates. For the normal time based sample rates, we assume the number of samples per minute is fixed. This is a perfectly fine assumption, except that Philip happens to have a dive with one extra sample at the 37th minute. We currently fail to parse this anomaly, because if we would continue the timestamps won't be strictly monotone anymore. In the worst case (e.g. more than one extra sample), time may move backwards.

This particular dive has a 15s sample rate, so normally you would see these times reported (where X is the timestamp in minutes, as stored in the data):

X:15 X:30 X:45 X:60 (= X+1:00)

X+1:15 X+1:30 X+1:45 X+1:60 (= X+2:00)

But if there is a 5th sample, you would see this:

X:15 X:30 X:45 X:60 (= X+1:00) X:75 (= X+1:15) <- Identical timestamp as the next sample!

X+1:15 X+1:30 X+1:45 X+1:60 (= X+2:00)

We could workaround this by using the same workaround as for the depth based sample rate, and distribute those 5 samples evenly (causing trouble for apps that assume a fixed sample rate). Or just ignore the presence of the timestamps in the data for dives with a time based sample rate. In that case the timestamp reported by libdivecomputer will go out of sync with the one stored in the data. This seems to be what Oceanlog does, although I wouldn't rely on it too much, because I have seen suspicious things there already (like having two samples with the exact same timestamps), and certainly with those depth based sample rates. No wonder they dropped that feature in the newer devices :-) Another alternative would be to just skip the extra samples, and pretend they are not there.

Jef

On 01/15/2012 07:50 PM, Dirk Hohndel wrote:

On Jan 15, 2012, at 8:45 AM, Jef Driesen wrote:

...

...

...

It's this timestamp that is causing the trouble in your case. Normally with a time based sample rate, there is a fixed number of samples per time unit. For example a 15s samplerate will always have 4 samples per minute. But that assumption is violated in your problematic dive. There are 5 samples with a timestamp of 37 minutes. I have absolutely no idea why the dive computer does this. It's the first time I encounter this.

But that shouldn't be a problem, if you then spread those evenly and give us an artificial 12 sec resolution. Subsurface makes no assumptions that the sample rate has to be constant.

The spreading only happens for the depth based sample rates. For the normal time based sample rates, we assume the number of samples per minute is fixed. This is a perfectly fine assumption, except that Philip happens to have a dive with one extra sample at the 37th minute. We currently fail to parse this anomaly, because if we would continue the timestamps won't be strictly monotone anymore. In the worst case (e.g. more than one extra sample), time may move backwards.

Instead of failing to parse, you could simply drop the extraneous sample - that would be much more user friendly, don't you think? After all, the rest of the data from the dive should be usable…

True. However that particular check was added under the assumption that it would never fail for real data. There is a similar check in case the timestamp would ever go backwards...

...

This particular dive has a 15s sample rate, so normally you would see these times reported (where X is the timestamp in minutes, as stored in the data):

X:15 X:30 X:45 X:60 (= X+1:00)

X+1:15 X+1:30 X+1:45 X+1:60 (= X+2:00)

But if there is a 5th sample, you would see this:

X:15 X:30 X:45 X:60 (= X+1:00) X:75 (= X+1:15)<- Identical timestamp as the next sample!

X+1:15 X+1:30 X+1:45 X+1:60 (= X+2:00)

We could workaround this by using the same workaround as for the depth based sample rate, and distribute those 5 samples evenly (causing trouble for apps that assume a fixed sample rate). Or just ignore the presence of the timestamps in the data for dives with a time based sample rate. In that case the timestamp reported by libdivecomputer will go out of sync with the one stored in the data. This seems to be what Oceanlog does, although I wouldn't rely on it too much, because I have seen suspicious things there already (like having two samples with the exact same timestamps), and certainly with those depth based sample rates. No wonder they dropped that feature in the newer devices :-) Another alternative would be to just skip the extra samples, and pretend they are not there.

Here's the moment where libDC gets to define the semantics and make them reasonable, and the software that uses libDC (MacDive, DivingLog, Subsurface) can then rely on that semantic. So once you release version 1.0, you should simply define that there is NO SUCH THING as a guarantee that the samples are evenly spaced and that instead the user of libDC has to take the time information in every sample into account.

In that case you can simply distribute the samples within that minute range (what a weird thing to do on Oceanic's part…). And if there are more than 60 of them, go to half second sampling rate (and simply announce that somewhere in the data structure).

This way whatever junk the dive computer throws at us, libDC converts it into something reasonable and consistent that the software can then rely on and use.

Bingo, everyone wins.

Don't worry, libdc has never guaranteed evenly spaced samples, and never will. Providing the time info is just more general and more future proof.

I doubt we'll ever see more than 60 samples a minute. If you manage to change depth at a speed of at least 5ft every second, I'm very impressed! But you never know of course. Sometimes the amount of junk can get frustrating :-)

Jef