The D<n> command is multiple bytes, right?
Yes. As I wrote, I tried the single char commands from the terminal software, and they did not work.
I think you are trying to re-use a bit too much from the backend that you used as an example. How to best read the response depends on how the protocol works. Is the length fixed or variable, do we known it in advance or not, etc.
Ok. As I may have mentioned in the beginning, this is totally unknown territory for me. From my understanding though, the code logic should be pretty simple: We do some base assumptions about buffer size, then attempt to read it full, and if the data from the device i less, then we should resize the buffer to match the actual data amount.
Going back here: When I successfully get the universal to send (even this seems to be unstable at times) the request byte, it does receive what seems to be an appropriate response.
But I think I understand your point, it could be that the chipset on the USB-cable pretending to be a serial device has some assumptions when the client is reading off it, and this has to be matched by the client software?
FWIW is it meaningful that the portmon log seems to indicate single byte read most of the time, or is it just an artifact of how portmon logs the communication?
Here's a snippet of the portmon log (I grep IRP_MJ for clarity):
87 0.00000698 ProLink2010oct. IRP_MJ_READ Silabser0 SUCCESS Length 3: 50 50 50 92 0.00001509 ProLink2010oct. IRP_MJ_CLEANUP Silabser0 SUCCESS 93 0.00320152 ProLink2010oct. IRP_MJ_CLOSE Silabser0 SUCCESS 94 0.02598515 ProLink2010oct. IRP_MJ_CREATE Silabser0 SUCCESS Options: Open 121 0.00000503 ProLink2010oct. IRP_MJ_READ Silabser0 SUCCESS Length 1: 50 130 0.00000531 ProLink2010oct. IRP_MJ_READ Silabser0 SUCCESS Length 1: 50 139 0.00000531 ProLink2010oct. IRP_MJ_READ Silabser0 SUCCESS Length 1: 50 145 0.00093783 ProLink2010oct. IRP_MJ_WRITE Silabser0 SUCCESS Length 1: 4D 155 0.00000559 ProLink2010oct. IRP_MJ_READ Silabser0 SUCCESS Length 1: 64 164 0.00000363 ProLink2010oct. IRP_MJ_READ Silabser0 SUCCESS Length 1: 0D 173 0.00000363 ProLink2010oct. IRP_MJ_READ Silabser0 SUCCESS Length 1: 0A 182 0.00000363 ProLink2010oct. IRP_MJ_READ Silabser0 SUCCESS Length 1: 76
0x50 equals P and seems to be the standby byte sent periodically.
So this equals to:
read: PPP read: P read: P read: P send: M read: d read: \r read: \n read: v ...
Poltsi
On 2016-01-04 14:10, Paul-Erik Törrönen wrote:
The D<n> command is multiple bytes, right?
Yes. As I wrote, I tried the single char commands from the terminal software, and they did not work.
I think you are trying to re-use a bit too much from the backend that you used as an example. How to best read the response depends on how the protocol works. Is the length fixed or variable, do we known it in advance or not, etc.
Ok. As I may have mentioned in the beginning, this is totally unknown territory for me. From my understanding though, the code logic should be pretty simple: We do some base assumptions about buffer size, then attempt to read it full, and if the data from the device i less, then we should resize the buffer to match the actual data amount.
A very important question is: how do you know the response has been received completely? Some communication protocols use fixed size packets, so you know the length in advance. If the length is variable, then very often it's communicated somewhere in the first bytes of the response. And if that's not the case, then you have to rely on a timeouts (e.g. you assume the response is complete if no byte arrives within a certain time). That's also the most error-prone method because you can't distinguish from a timeout due to some error.
Going back here: When I successfully get the universal to send (even this seems to be unstable at times) the request byte, it does receive what seems to be an appropriate response.
I'm not sure if this is relevant, but I've noticed that many dive computers need some time between setting up the serial line, and starting the communication by sending bytes. If you'll look at the other backends, you'll notice many sleep calls with a delay of several hundred milliseconds. So that's certainly worth trying.
But I think I understand your point, it could be that the chipset on the USB-cable pretending to be a serial device has some assumptions when the client is reading off it, and this has to be matched by the client software?
Probably not the usb-serial chipset, but the dive computer software (or hardware) behind it. The usb-serial chipset is used in many different applications. It only needs to care about being compatible with RS-232 on one side and USB on the other side. But the dive computer can do whatever it wants too.
FWIW is it meaningful that the portmon log seems to indicate single byte read most of the time, or is it just an artifact of how portmon logs the communication?
Portmon has nothing to do with this. It's the application that is reading single bytes.
Here's a snippet of the portmon log (I grep IRP_MJ for clarity):
87 0.00000698 ProLink2010oct. IRP_MJ_READ Silabser0 SUCCESS Length 3: 50 50 50 92 0.00001509 ProLink2010oct. IRP_MJ_CLEANUP Silabser0 SUCCESS 93 0.00320152 ProLink2010oct. IRP_MJ_CLOSE Silabser0 SUCCESS 94 0.02598515 ProLink2010oct. IRP_MJ_CREATE Silabser0 SUCCESS Options: Open 121 0.00000503 ProLink2010oct. IRP_MJ_READ Silabser0 SUCCESS Length 1: 50 130 0.00000531 ProLink2010oct. IRP_MJ_READ Silabser0 SUCCESS Length 1: 50 139 0.00000531 ProLink2010oct. IRP_MJ_READ Silabser0 SUCCESS Length 1: 50 145 0.00093783 ProLink2010oct. IRP_MJ_WRITE Silabser0 SUCCESS Length 1: 4D 155 0.00000559 ProLink2010oct. IRP_MJ_READ Silabser0 SUCCESS Length 1: 64 164 0.00000363 ProLink2010oct. IRP_MJ_READ Silabser0 SUCCESS Length 1: 0D 173 0.00000363 ProLink2010oct. IRP_MJ_READ Silabser0 SUCCESS Length 1: 0A 182 0.00000363 ProLink2010oct. IRP_MJ_READ Silabser0 SUCCESS Length 1: 76
0x50 equals P and seems to be the standby byte sent periodically.
So this equals to:
read: PPP read: P read: P read: P send: M read: d read: \r read: \n read: v ...
This is right after opening the serial port, so it could be that the dive computer is telling you "I'm ready to receive commands". Maybe it keeps sending this byte until it receives a command? This might also be the reason why the communication is unstable: If you are trying to send too fast, the dive computer isn't ready yet, and your command might got lost.
Jef
A very important question is: how do you know the response has been received completely? Some communication protocols use fixed size packets, so you know the length in advance. If the length is variable, then very often it's communicated somewhere in the first bytes of the response. And if that's not the case, then you have to rely on a timeouts (e.g. you assume the response is complete if no byte arrives within a certain time). That's also the most error-prone method because you can't distinguish from a timeout due to some error.
Looking at the portmon dumps, there are no explicit values passed which would tell the length of the requested data. There are two lines in the header, Mem and Memi which both carry multiple numerical values, but I have not comprehended their function yet.
My suspicion is that for the header part it is 'known' in the sense that it has certain lines.
As for the dive telemetry, it begins with a 'Profile'-line, and ends with a 'End'-line followed by '@@'-line. The lines are dos-terminated, ie. \r\n.
As you say, this is quite inaccurate, and I don't think even the original software always manage to handle this correctly.
I'm not sure if this is relevant, but I've noticed that many dive computers need some time between setting up the serial line, and starting the communication by sending bytes. If you'll look at the other backends, you'll notice many sleep calls with a delay of several hundred milliseconds. So that's certainly worth trying.
Ok, I'll try with putting in some sleeps between the initialization of the serial connection, and requesting a dump.
FWIW is it meaningful that the portmon log seems to indicate single byte read most of the time, or is it just an artifact of how portmon logs the communication?
Portmon has nothing to do with this. It's the application that is reading single bytes.
What I meant was that I can presume that I should be reading just one byte at a time in the dump-function, instead of trying to use dynamic buffers as I do now in the function.
So this equals to: read: PPP read: P read: P read: P send: M read: d read: \r read: \n read: v ...
This is right after opening the serial port, so it could be that the dive computer is telling you "I'm ready to receive commands". Maybe it keeps sending this byte until it receives a command?
'this byte' would be P. And once I reply with the M-byte, I start receiving the header data. The response should start with a 'd\r\n' when the header part is returned, so the above would look like:
d ver=V009B Recint=5 SN=4AAD12245D5B7038 V009B Mem 0, 4000, 1192 ...
This might also be the reason why the communication is unstable: If you are trying to send too fast, the dive computer isn't ready yet, and your command might got lost.
I don't think so. I do receive the 'wait' byte (ie. 'P'), and my response is a single byte ('M').
I've uploaded the Portmon logs as well as the actual downloaded data here:
http://poltsi.fi/Sukeltaminen/Sentinel/portmon-logs.tar.bz2
The portmon logs are in the main directory and are named as <device>-<unit number>-<date>-<purpose>-<attempt>.log
My unit# is 328, but I also borrowed the #007 from my friend, unfortunately his unit did not have the PC link PIN, so I was unable to download actual dive data from that unit. Also to note is that the electronics was switched on my unit, so between November and December the firmware version (ver=) as well as serial number (SN=) changed. The purpose is loosely at what point of actions the recording is. Connect is when I start the Prolink software and it connects to the rebreather, download is when I explicitly select to download either a single dive, or all new dives. The attempt is a running number, as I noticed only later that I need to restart portmon in order to properly get separate logs for different actions.
There are 3 sub-directories:
dump-parsed pm-parsed prolink-parsed
Dump-parsed is the output from my script (dump_parser.sh) of each portmon-log, and contains the WRITE/READ data. pm-parsed is the output from your parser, the directory contains separate read and write-files for each portmon-log. prolink-parsed contains the downloaded data stored by ProLink, which is the original software. Since my portmon-logs may contain the download of several dives, I have named them as <portmon filename>-n.log.
Amd of course the parsing scripts themselves.
I noticed that the last portmon-log did not put the actual read data on the same line as the IRP_MJ_READ, instead it looked like this:
68 0.00000000 ProLink2010oct. IRP_MJ_READ Silabser0 Length 1 68 0.00000559 SUCCESS Length 1: 50
Even if I had not touched the settings of portmon between this and the previous logging.
There is also something odd going on with the pm-based parser, for example the Sentinel-328-20160104-Download-001.log-read is only 14k, while the actual data is almost 10x that, both in the prolink-parsed as well as dump-parsed.
Poltsi
On 2016-01-10 00:39, Paul-Erik Törrönen wrote:
A very important question is: how do you know the response has been received completely? Some communication protocols use fixed size packets, so you know the length in advance. If the length is variable, then very often it's communicated somewhere in the first bytes of the response. And if that's not the case, then you have to rely on a timeouts (e.g. you assume the response is complete if no byte arrives within a certain time). That's also the most error-prone method because you can't distinguish from a timeout due to some error.
Looking at the portmon dumps, there are no explicit values passed which would tell the length of the requested data. There are two lines in the header, Mem and Memi which both carry multiple numerical values, but I have not comprehended their function yet.
My suspicion is that for the header part it is 'known' in the sense that it has certain lines.
As for the dive telemetry, it begins with a 'Profile'-line, and ends with a 'End'-line followed by '@@'-line. The lines are dos-terminated, ie. \r\n.
As you say, this is quite inaccurate, and I don't think even the original software always manage to handle this correctly.
I suspect that their software is reading the response line by line, using the \r\n sequence as the end-of-line marker. And then that 'End' followed by '@@' as the end-of-message marker. That probably also explains why they are reading single bytes: they have to inspect every byte to find for those markers.
What I meant was that I can presume that I should be reading just one byte at a time in the dump-function, instead of trying to use dynamic buffers as I do now in the function.
Reading one byte at a time isn't the most efficient, but if it's necessary then fine. You'll still need a dynamic buffer, because you don't know how many bytes you'll receive. Just append each byte, and the buffer will grow automatically.
So this equals to: read: PPP read: P read: P read: P send: M read: d read: \r read: \n read: v ...
This is right after opening the serial port, so it could be that the dive computer is telling you "I'm ready to receive commands". Maybe it keeps sending this byte until it receives a command?
'this byte' would be P. And once I reply with the M-byte, I start receiving the header data. The response should start with a 'd\r\n' when the header part is returned, so the above would look like:
d ver=V009B Recint=5 SN=4AAD12245D5B7038 V009B Mem 0, 4000, 1192 ...
This might also be the reason why the communication is unstable: If you are trying to send too fast, the dive computer isn't ready yet, and your command might got lost.
I don't think so. I do receive the 'wait' byte (ie. 'P'), and my response is a single byte ('M').
How does the response looks like when it fails? Do you still get the P bytes?
I've uploaded the Portmon logs as well as the actual downloaded data here:
http://poltsi.fi/Sukeltaminen/Sentinel/portmon-logs.tar.bz2
The portmon logs are in the main directory and are named as <device>-<unit number>-<date>-<purpose>-<attempt>.log
My unit# is 328, but I also borrowed the #007 from my friend, unfortunately his unit did not have the PC link PIN, so I was unable to download actual dive data from that unit. Also to note is that the electronics was switched on my unit, so between November and December the firmware version (ver=) as well as serial number (SN=) changed. The purpose is loosely at what point of actions the recording is. Connect is when I start the Prolink software and it connects to the rebreather, download is when I explicitly select to download either a single dive, or all new dives. The attempt is a running number, as I noticed only later that I need to restart portmon in order to properly get separate logs for different actions.
There are 3 sub-directories:
dump-parsed pm-parsed prolink-parsed
Dump-parsed is the output from my script (dump_parser.sh) of each portmon-log, and contains the WRITE/READ data. pm-parsed is the output from your parser, the directory contains separate read and write-files for each portmon-log. prolink-parsed contains the downloaded data stored by ProLink, which is the original software. Since my portmon-logs may contain the download of several dives, I have named them as <portmon filename>-n.log.
Amd of course the parsing scripts themselves.
I noticed that the last portmon-log did not put the actual read data on the same line as the IRP_MJ_READ, instead it looked like this:
68 0.00000000 ProLink2010oct. IRP_MJ_READ Silabser0 Length 1 68 0.00000559 SUCCESS Length 1: 50
Even if I had not touched the settings of portmon between this and the previous logging.
I've never noticed this before. Strange. Still will certainly causes issues for my scripts.
There is also something odd going on with the pm-based parser, for example the Sentinel-328-20160104-Download-001.log-read is only 14k, while the actual data is almost 10x that, both in the prolink-parsed as well as dump-parsed.
It's caused by another "error" in the portmon file. Normally read operations contain "Length n: XX YY ZZ" in the 7th column. But in your files there are a few with just "Length 1". My sed regex doesn't recognizes that pattern and leaves it in, and that causes trouble when xxd tries to convert the data back to binary.
Modify the sed line in the pm-simplify-rw script as follows:
sed -e 's/IRP_MJ_//' -e 's/IOCTL_SERIAL_//' -e 's/Length [0-9]*//' -e 's/: //' | \
Jef
I dropped the rEvo from the subject since it has nothing to do with the discussion at hand.
I suspect that their software is reading the response line by line, using the \r\n sequence as the end-of-line marker. And then that 'End' followed by '@@' as the end-of-message marker. That probably also explains why they are reading single bytes: they have to inspect every byte to find for those markers.
Yeah, and I've updated now the github repository with a version that does something similar (when using the universal-command).
FWIW I noticed that not all my changes were pushed to github, git is also new to me and I need to remember to do the 'git push' after the 'git commit...'.
So what the vms_sentinel_device_dump currently does is send 'M' to the rebreather, which should get a response of listing all the dive headers. I read the data one byte at a time, and look at the end of the data-buffer in search for a) line ends '\r\n', dive header separators 'd\r\n' as well as end of the whole list '@@P'. It seems to do a reasonable job of this.
I run the following command:
examples/universal -b sentinel -l ../logs/sentinel-`date '+%Y-%m-%d-%H%M%S'`.log -d ../logs/dives-`date '+%Y-%m-%d-%H%M%S'`.log /dev/ttyUSB0 | tee ../logs/output-`date '+%Y-%m-%d-%H%M%S'`.log
And I've uploaded the logs of the last successful run here:
http://poltsi.fi/Sukeltaminen/Sentinel/sentinel_logs-2016-01-30-000659.tar.g...
At the end of the output-file (search for 'Data is') I print out the data-buffer which looks pretty ok
How does the response looks like when it fails? Do you still get the P bytes?
It only continues sending the P character.
What is the real purpose of the *_device_dump?
Currently I have used it as a convenient playground for my exploration, as it is called from the universal, but I guess it has some real function too?
I have insofar not been able to detect any command that would dump everything in the rebreather memory.
I guess the next step would be to be to actually store the list of dives somewhere, but it's not clear to me where that happens?
Since I used cressi_leonard as a base, it no longer works as a template since the devices differ too much.
Poltsi
-
"Paul-Erik Törrönen" -
Jef Driesen