Hi All,
I've been staring at and playing with this data for a week now and I don't feel any closer to deciphering the dive profile sample data from my Cochran EMC-20H.
Personally I love a challenge like this and I'm hoping some of you do as well because this one is tough. Instead of digging into a good book this weekend why not sit back with a laptop or tablet and this puzzle.
You'll find everything you need at the link below. I've removed the mundanely easy work from this puzzle. Included are binary files, annotated hexdumps to illustrate the data and some programs to extract and display data (don't judge, these are quick and dirty programs.) Start with the README and you'll be into the fun part within minutes.
https://drive.google.com/file/d/0B-VD5cn7zgGHN1pVRWpib2pJc0k/edit?usp=sharin...
On Fri, Jun 27, 2014 at 5:46 PM, John Van Ostrand john@vanostrand.com wrote:
Hi All,
I've been staring at and playing with this data for a week now and I don't feel any closer to deciphering the dive profile sample data from my Cochran EMC-20H.
Personally I love a challenge like this and I'm hoping some of you do as well because this one is tough. Instead of digging into a good book this weekend why not sit back with a laptop or tablet and this puzzle.
You'll find everything you need at the link below. I've removed the mundanely easy work from this puzzle. Included are binary files, annotated hexdumps to illustrate the data and some programs to extract and display data (don't judge, these are quick and dirty programs.) Start with the README and you'll be into the fun part within minutes.
https://drive.google.com/file/d/0B-VD5cn7zgGHN1pVRWpib2pJc0k/edit?usp=sharin...
I've made some progress on this but I could still use another set of eyes.
Each sample is in 3 byte structure and the composition of that structure changes every second between two types. The odd seconds (if it starts at 1 sec) have a structure that seems to have a pressure change in mbar, the even seconds have a structure that seems to store temperature samples. Both types of structures seem to hold tissue compartment estimates.
Two seconds of samples looks like this (hex and binary for each sample shown):
Odd Seconds (bytes 1 to 3) 0x02 00000010 Bit 7, when 0, indicates that a sample follows. Bits 6 is a flag, maybe + or - indicator or data, bits 5-3 are always 0, bits 2-0 vary often. 0x14 00010100 Bit 7 indicates a negative value in this byte, bits 6-0 seem to be mbar increments from the last sample. 0x03 00000011 Byte indicates tissue compartment loading for one compartment.
Even Seconds (bytes 4 to 6) 0x01 00000001 Bit 7, when 0, indicates that a sample follows. Bits 6 is a flag, maybe + or - indicator or data, bits 5-3 are always 0, bits 2-0 vary often. 0x30 00110000 Bits 6-0 change very slow, like a temperature sample but the value is off and may need to be adjusted, maybe to a calibration. 0x02 00000010 Byte indicates tissue compartment loading for one compartment.
It's clear from using the dive software that samples for depth and temp are displayed for each second yet it appears that depth and temp are only recorded every other second. I've extracted what appears to be the mbar increment data bits and graphed them. They closely, but not exactly, follow a dive profile from the vendor software. There is detail missing and the computed depths are a few feet off. I may not have mbar to feet conversion right or there is a calibration I need to consider.
I suspect that bytes 1 and 4 hold sample data as well but there aren't enough free bits to match the 7 bits used for depth and the 7 bits seemingly used for temperature.
I think the bits may be scattered across more than one sample too.
There are 20 tissues tracked by the DC and the samples are spread across 20 seconds of depth/temp samples. That set of 20 samples is separated from the next set with 4 seconds of depth/temp samples except instead of tissue loading there is different data. Where tissue data would be are either FF or 00. I don't recall seeing the 00s change but the FFs change about mid dive and most bits go to 0. On a repetitive dive, a deeper wreck in this case, the FFs never showed up. It may be tissue loading, or OTUs or some other data.
Because of all this scattering of samples I'm suspecting that there is a temperature increment in the depth sample and a depth increment in the temp sample. The temp increment could be simple, just a -1 or +1 indicating a change in temperature. I've only looked briefly but I can't see which bits it could be. There aren't enough bits left over to make 7 bits of mbar increment so maybe the even-second depth sample is an increment of the odd-second depth sample.
In thinking this through to a practical application because maybe I'm including bits in the LSB portion that aren't part of the sample. The software displays increments of 0.25 feet, that's about 8 mbar suggesting a minimum increment and a four foot per second change in depth might be a reasonable limit giving a max increment of about 128. I suppose this could be done with 5 bits (a +/- bit and 4 data bits) but it appears to be done with 8 in the main depth sample.
I'm done for the evening and I'll be driving or diving all day tomorrow.
I'd be happy to discuss this later this week if anyone is also working on this.
On 2014-06-30 04:37, John Van Ostrand wrote:
Each sample is in 3 byte structure and the composition of that structure changes every second between two types. The odd seconds (if it starts at 1 sec) have a structure that seems to have a pressure change in mbar, the even seconds have a structure that seems to store temperature samples. Both types of structures seem to hold tissue compartment estimates.
Two seconds of samples looks like this (hex and binary for each sample shown):
Odd Seconds (bytes 1 to 3) 0x02 00000010 Bit 7, when 0, indicates that a sample follows. Bits 6 is a flag, maybe + or - indicator or data, bits 5-3 are always 0, bits 2-0 vary often. 0x14 00010100 Bit 7 indicates a negative value in this byte, bits 6-0 seem to be mbar increments from the last sample. 0x03 00000011 Byte indicates tissue compartment loading for one compartment.
Even Seconds (bytes 4 to 6) 0x01 00000001 Bit 7, when 0, indicates that a sample follows. Bits 6 is a flag, maybe + or - indicator or data, bits 5-3 are always 0, bits 2-0 vary often. 0x30 00110000 Bits 6-0 change very slow, like a temperature sample but the value is off and may need to be adjusted, maybe to a calibration. 0x02 00000010 Byte indicates tissue compartment loading for one compartment.
It's clear from using the dive software that samples for depth and temp are displayed for each second yet it appears that depth and temp are only recorded every other second.
Based on the above description (I haven't looked at your code), I think there is only one 6 byte large sample, rather than two different but alternating samples. In that case the sample rate is of course two seconds instead of one. The application may still interpolate (or copy) samples to give the impression of a one second sample rate.
With fixed size samples, I suggest you make a hexdump with one sample per line. Because bytes in the same column usually always represent the same type of data in a fixed size sample, you'll more easily notice certain patterns in the data. It certainly helped me to figure out the data structure in a several cases.
I've extracted what appears to be the mbar increment data bits and graphed them. They closely, but not exactly, follow a dive profile from the vendor software. There is detail missing and the computed depths are a few feet off. I may not have mbar to feet conversion right or there is a calibration I need to consider.
They might take into account atmospheric pressure and/or salinity. If there is a fixed absolute different, then there is a different atmospheric correction. If it's a fixed scale factor, then that's likely a different salinity factor.
Another reason could be rounding errors and scaling factors. If there are for example unit conversions necessary, and the application only shows a limited number (or no) digits than the values may appear to be slightly off. But in the end that's just a rounding issue. Scaling factors are also very common. If data is for example stored in units of 1/16 feet, the magnitude values will be off by a factor 16 until you figure out this scaling factor.
Jef
On Mon, Jun 30, 2014 at 2:20 AM, Jef Driesen jef@libdivecomputer.org wrote:
I've extracted what appears to be the mbar increment data bits and graphed them. They closely, but not exactly, follow a dive profile from the vendor software. There is detail missing and the computed depths are a few feet off. I may not have mbar to feet conversion right or there is a calibration I need to consider.
They might take into account atmospheric pressure and/or salinity. If there is a fixed absolute different, then there is a different atmospheric correction. If it's a fixed scale factor, then that's likely a different salinity factor.
Also, how sure are you (John) that the sample data is actually in millibar?
It could easily be in actual depth - in cm. The pressure of one cm of water is very close to one mbar, so the two are almost interchangeable, but it would be a difference of a couple of percent - so a foot or two off depending on depth. And from what I've seen, most dive computers do tend to sample actual depth, not pressure, if only because that is what they show on the screen. Logically, "pressure" is what the sensors give you, and what really matters for deco calculations, so converting to "depth" is fraught with problems (surface pressure, salf-vs-fresh, yadda yadda), but it's (a) what people want to see and (b) in the end, the errors of a percent or two are not really material, so ..
But yes, as Jef says, it could be any number of other factors too. The difference between salt/fresh water is a few percent too, and quite frankly, having seen the mess some dive log software make of things, just being *wrong* can be a few percent too (ie not actually using proper calculations at all, ie just using the rough "1 atm is 10 m of depth", or just mixing up atm and bar). We've had those kinds of errors too, and judging from what I've seen, we've definitely had _less_ than some.
Linus
On Mon, Jun 30, 2014 at 11:30 AM, Linus Torvalds < torvalds@linux-foundation.org> wrote:
On Mon, Jun 30, 2014 at 2:20 AM, Jef Driesen jef@libdivecomputer.org wrote:
I've extracted what appears to be the mbar increment data bits and graphed them. They closely, but not exactly, follow a dive profile from the vendor software. There is detail missing and the computed depths are a few feet off. I may not have mbar to feet
conversion
right or there is a calibration I need to consider.
They might take into account atmospheric pressure and/or salinity. If
there
is a fixed absolute different, then there is a different atmospheric correction. If it's a fixed scale factor, then that's likely a different salinity factor.
FWIW this is fresh water which also means atmospheric pressure is below 1 atm. It's somewhere between 70 and 700 feet depending on dive site. I'll have to look at how much pressure change that is but the DC reports 0 Kft of altitude.
Also, Jeff, I considered that it might be interpolating depth to make 1 sec samples so I looked in the vendor's own software for an example of data where that couldn't have been the case. I found an example of a "W" shaped profile where the depth was up, down, up, down at 1 second intervals. I don't know how a two second sample rate and interpolation could produce that.
Also, how sure are you (John) that the sample data is actually in millibar?
It could easily be in actual depth - in cm. The pressure of one cm of
water is very close to one mbar, so the two are almost interchangeable, but it would be a difference of a couple of percent - so a foot or two off depending on depth. And from what I've seen, most dive computers do tend to sample actual depth, not pressure, if only because that is what they show on the screen. Logically, "pressure" is what the sensors give you, and what really matters for deco calculations, so converting to "depth" is fraught with problems (surface pressure, salf-vs-fresh, yadda yadda), but it's (a) what people want to see and (b) in the end, the errors of a percent or two are not really material, so ..
It could be centimetres but because this is fresh water wouldn't that be identical to millibar? The computer has external contacts and can sense between fresh and salt water. I may try diving the computer in a zip-lock bag full of salt water aquarium water to see how it logs salt water dives. I suspect that the dive start structure will have a salt/fresh water flag so the vendor software can display depth.
The difference on a dive to 27 ft is about 5ft, that's quite significant by itself and the difference grows as the depth does.
But yes, as Jef says, it could be any number of other factors too. The difference between salt/fresh water is a few percent too, and quite frankly, having seen the mess some dive log software make of things, just being *wrong* can be a few percent too (ie not actually using proper calculations at all, ie just using the rough "1 atm is 10 m of depth", or just mixing up atm and bar). We've had those kinds of errors too, and judging from what I've seen, we've definitely had _less_ than some.
It's confusing me to the point that I'm looking for other clues for speculation. One clue may be that the vendor software exports data rounded to what looks like 0.25 increments. So depths of 0.3, 0.5, 0.8 are seen but in the vendor's graphs the depths use two decimal digits and are 0.25 increments but shifted. One profile shows data like 0.07, 0.32, 0.57, 0.82 and another shows 0.18, 0.43, 0.68, 0.93. There's a minor shift happening. Temperature? Altitude?
Maybe the most productive idea would be to get the data from the .CAN file in the hopes it's stored in a more accurate form. Linus you were looking for help deciphering a CAN file a while ago. Did you ever find the depth data? I've taken a cursory look and I can see that the start of the file has offsets into the data, each offset looks like a dive.
On Wed, Jul 2, 2014 at 10:39 AM, John Van Ostrand john@vanostrand.com wrote:
It could be centimetres but because this is fresh water wouldn't that be identical to millibar?
No, it's the other way, actually.
The thing is, one cubic cm of pure water may weigh pretty much exactly one gram, and that makes some metric people think that fresh water gives you the nice straight "10m is one bar" thing, because they assume that's how it all works out. But that's not actually true.
Bar being a measure of pressure, it's not the *mass* of water that matters, but the *weight*. And that has another factor in it entirely: the gravity of earth, giving a constant factor of 9.81.
So the normal "10m of water is on ATM" (note: ATM, not bar) is actually not a result of metric powers-of-ten at all, but rather it's a happenstance of the weigth of *SALT* water (roughly 1.03g per cubic cm), and the gravity of earth (9.81m/s2), and one ATM being slightly bigger than 1 bar.
Check it out: - one ATM = 1.013 bar - 9.81*1.03 = 1.010
and then it just so happens that the pressure at 10m ends up being almost exactly 1 ATM more than surface pressure.
But in fresh water, you don't get the "1.03" factor, and you end up having the pressure of 10m of water being about 981 mbar, ie 0.96 atm. No round numbers anywhere.
The computer has external contacts and can sense between fresh and salt water. I may try diving the computer in a zip-lock bag full of salt water aquarium water to see how it logs salt water dives. I suspect that the dive start structure will have a salt/fresh water flag so the vendor software can display depth.
Interesting, I don't think I've seen (or been aware of) dive computers that sense the salinity automatically.
Personally, I think salinity sensing is kind of stupid, actually. It has no serious meaning, since the actual deco calculations do not care at all. The sensor senses pressure, and that's the only thing that really matters. The conversion to depth is "pointless" - it has no real meaning for diving, except if you happen to have other depth measurements that you want to match with, which is very very rare.
So the only reason to give depth at all is that it's what people "expect", but since people don't actually care about the roughly 3% "real absolute difference" in depth in any normal situation, why even bother changing the calculations? Even if you know the depth of the dive site, on most diving things like tides etc tend to be more than the 3% difference.
So I think "salt vs fresh" is interesting perhaps from a logging standpoint - the same way that it can be interesting to differentiate a boat dive from a shore dive. But I think it's over-engineering to take it into account for depth calculations.
(But hey, maybe cochran users care, because they go deeper, and are potentially matching up their dive depth against some other measurement like sonar depths from a boat. At that point, you *might* care. I find it unlikely, but whatever)
The difference on a dive to 27 ft is about 5ft, that's quite significant by itself and the difference grows as the depth does.
Ok, that's a much bigger difference than I thought, so there's clearly something else going on. The "couple of percent" differences you can get from the whole salt-vs-fresh and atm-vs-bar confusions are *much* smaller than that 5ft in 27ft.
Maybe the most productive idea would be to get the data from the .CAN file in the hopes it's stored in a more accurate form. Linus you were looking for help deciphering a CAN file a while ago. Did you ever find the depth data? I've taken a cursory look and I can see that the start of the file has offsets into the data, each offset looks like a dive.
I never got anywhere with the cochran data, no.
Linus
On 2014-07-02 20:03, Linus Torvalds wrote:
On Wed, Jul 2, 2014 at 10:39 AM, John Van Ostrand john@vanostrand.com wrote:
The computer has external contacts and can sense between fresh and salt water. I may try diving the computer in a zip-lock bag full of salt water aquarium water to see how it logs salt water dives. I suspect that the dive start structure will have a salt/fresh water flag so the vendor software can display depth.
Interesting, I don't think I've seen (or been aware of) dive computers that sense the salinity automatically.
I'm also not aware of any dive computer that supports this. The water contacts are usually only used to activate the dive computer's dive mode. Technically, it might be possible to measure the salinity by measuring the conductivity of the water. But as Linus already explained, the dive computer doesn't need this information for its decompression calculations. So why even bother to measure it? Dive computers have other and much more important things to do.
Personally, I think salinity sensing is kind of stupid, actually. It has no serious meaning, since the actual deco calculations do not care at all. The sensor senses pressure, and that's the only thing that really matters. The conversion to depth is "pointless" - it has no real meaning for diving, except if you happen to have other depth measurements that you want to match with, which is very very rare.
So the only reason to give depth at all is that it's what people "expect", but since people don't actually care about the roughly 3% "real absolute difference" in depth in any normal situation, why even bother changing the calculations? Even if you know the depth of the dive site, on most diving things like tides etc tend to be more than the 3% difference.
So I think "salt vs fresh" is interesting perhaps from a logging standpoint - the same way that it can be interesting to differentiate a boat dive from a shore dive. But I think it's over-engineering to take it into account for depth calculations.
I think the only case where a difference of a few percentages might matter somewhat is when doing (extreme) deep dives with a dive computer in gauge mode and tables. In that case you do use the calculated depth for the tables, and not the pressure measured by the sensor.
Jef
Good news everyone!
I've finally figured it out, at least close enough that I can claim victory.
It's interesting how an idea has inertia. I had originally thought I had isolated the depth bits because they were the most dynamic bits. I was stuck thinking those were depth until I realized they could have been depth change values. I'm happy to say I wasn't far off, they were ascent rate values in feet, sampled every two seconds with intermediate samples interpolated.
The bits I that turned out to be depth change were so few that I had not even considered them for depth and I never reconsidered them when I started working on depth change, until today.
The depth change samples are 7 bits of depth change in 0.25 foot increments allowing a max change of 15.75 ft/second. My depth changes have all been less than 1.5 f/s so I haven't had a depth change that uses those higher bits, which are always 0.
What I'm still missing is the starting position. All depth change samples I've looked at over several dives have fit *exactly* what the vendor software reports however I haven't found the starting depth. It's over 6' because of how the computer works.
The temperature is odd but I have it accurate. It's in degrees F from 20F.
The only thing missing is that i have 32 bytes in a grouping of 24 samples that are unallocated to any sample. I can probably ignore them.
Also,
Part of my problem was I was overthinking Cochran. The computer is quite sophisticated and it clearly feels designed by an innovator and engineer who might try to stuff bits anywhere. It has rough edges so I doubt any usability expert or ergonomics person has had any input. And, yes, it does sense salt/fresh water.
I purchased the Cochran EMC-20H after a recommended from a friend of a friend and after the friend purchased and dove it while we were both in the tropics. It was far more liberal than his Mares Icon and my Suunto Vyper II giving more than 10 minutes of NDL over my dive computer on dives later in the day. It has 50 increments of conservatism. Zero seems to be Navy Tables (on the first dive) while 20 seems to be close to PADI tables and 45 is close to Suunto.
After looking at the material it seemed like the Cochran algorithm is for engineers where algos like the Suunto and Mares are for lawyers. Well that or Mike Cochran has learned how to alter physics or dive physiology. Maybe it just draws a better "Texas bullseye" around the test dive data.
The company claims that they were first to release many features and are still the only DC that offers some features like workload compensation (based on gas consumption), water temperature compensation (colder = more metabolically active), a flight data recorder in the DC (giving a N2 credit for diving after flying), and the ability to clone tissue loads to other dive computers.
Now in reality not much of that is very useful to me but I like the idea that they are doing what I thought DCs should do, and that was 8 years ago after OW cert. But 18 navies around the world are attracted to those features, that's impressive to me.
The one big reason I discounted bright and shiny computers like Shearwater, Liquivision and OSTC was the battery life. I have enough rechargeables to deal with in my life the last thing I need is to arrive at a dive site with a dead battery. The Cochran batteries last 2 years or 1000 dive hours, that bullet point is near the top of the list for me.
On Wed, Jul 2, 2014 at 6:39 PM, John Van Ostrand john@vanostrand.com wrote: [...]
Also, Jeff, I considered that it might be interpolating depth to make 1 sec samples so I looked in the vendor's own software for an example of data where that couldn't have been the case. I found an example of a "W" shaped profile where the depth was up, down, up, down at 1 second intervals. I don't know how a two second sample rate and interpolation could produce that.
Maybe contents of the publication below could help? In the abstract they mention depth data recording every 1s (it is just a shot, I have no full access to it).
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=968236
Also, I can remember some publication by Thalmann (?) containing source code snippets for a dive computer software. Cannot find it now. :/
All the publications about US Navy dive computer link Cochran to Thalmann ideas about dive computers and decompression models (lawyers could disagree ;), i.e. see "Physiology and Medicine of Diving" (page 465-466 in my copy of the book, you can find older version of the chapter at [1], logical page 463). Also in [2]. Wikipedia might jump you into more publications.
It is all in the context of VVAL18 and whatever your dive computer uses now is newer, but I suspect the data logging scheme is probably very similar if not the same.
I hope above will not put you into a wrong direction.
[1] https://www.us.elsevierhealth.com/media/us/samplechapters/9780702025716/9780... [2] http://archive.rubicon-foundation.org/xmlui/bitstream/handle/123456789/10147...
[...]
Regards,
w
-
Artur Wroblewski -
Jef Driesen -
John Van Ostrand -
Linus Torvalds