diff --git a/drafts/StreamTransportS.odt b/drafts/StreamTransportS.odt index a239b1b4..3e1aa7a4 100644 Binary files a/drafts/StreamTransportS.odt and b/drafts/StreamTransportS.odt differ diff --git a/drafts/StreamTransportS.pdf b/drafts/StreamTransportS.pdf index 772e53bd..9f7b70db 100644 Binary files a/drafts/StreamTransportS.pdf and b/drafts/StreamTransportS.pdf differ diff --git a/drafts/StreamTransportTN.odt b/drafts/StreamTransportTN.odt index 472bfba9..dee5d104 100644 Binary files a/drafts/StreamTransportTN.odt and b/drafts/StreamTransportTN.odt differ diff --git a/drafts/StreamTransportTN.pdf b/drafts/StreamTransportTN.pdf index 36eb1d15..d42ca7a8 100644 Binary files a/drafts/StreamTransportTN.pdf and b/drafts/StreamTransportTN.pdf differ diff --git a/drafts/generated/StreamTransportS.txt b/drafts/generated/StreamTransportS.txt index 3f158006..a2164895 100644 --- a/drafts/generated/StreamTransportS.txt +++ b/drafts/generated/StreamTransportS.txt @@ -9,19 +9,22 @@  Streaming transport can also be used when the data transfer is unknown or indefinite in its volume or persistence. - References and Context (Normative) + 3 References and Context (Normative) This specification is in the context of the following OpenLCB Standards: + * The OpenLCB Unique Identifiers Standard, which defines the method for allocating unique identifiers + that form the basis of Stream Content UIDs, defined below. + * The OpenLCB Message Network Standard, which defines the basic messages and how they interact. Higher-level protocols are based on this message network, but are defined elsewhere. The Message - Network Standard defines the encoding of stream data message on CAN physical layer, which is - referenced here. + Network Standard defines the encoding of stream data messages on the CAN physical layer, which is + referenced here. The Message Network Standard also defines the error codes that are used here. * The OpenLCB Memory Access Configuration Protocol Standard, which will be used as an example of an application-level protocol that interacts with the Streaming protocol for data transport. - 3 Definitions (Normative) + 4 Definitions (Normative) Stream is a uni-directional sequence of bytes transmitted from one OpenLCB node to another OpenLCB node. There is no out-of-band information that can be sent along with the payload using the Streaming @@ -32,27 +35,27 @@ Destination node is the OpenLCB node which is receiving the data. - Source Stream ID (SID) is a one-byte (8-bit) identifier (range [0..254], reserved [ 255]), assigned by + Source Stream ID (SID) is a one-byte (8-bit) identifier (range [0..254], reserved [255]), assigned by the source node at its discretion to identify the stream. - Destination Stream ID (DID) is a one-byte (8-bit) identifier (range [0..254], reserved [ 255]), assigned + Destination Stream ID (DID) is a one-byte (8-bit) identifier (range [0..254], reserved [255]), assigned by the destination node at its discretion to identify the stream. Stream Content UID is a non-zero 6-byte globally unique identifier assigned to specify the type (format - and purpose) of the data being transmitted. Stream Content UIDs are assigned in a similar fashion as - Node IDs, but from a different numeric space. + and purpose) of the data being transmitted. Stream Content UIDs are assigned using Unique Identifiers in + the same fashion as Node IDs. (Max) Buffer size is an unsigned 2-byte value (range [1..65535]) specifying the maximum number of bytes in-flight; that is, the maximum number of bytes the source node may send without receiving a reception acknowledgement from the destination node. - 4  Message Formats + 5  Message Formats In the following, the “common MTI” column specifies the MTI value to be used when communicating in OpenLCB common format.  The Common MTI is an abstract numeric description intended as a normative guide to the construction of concrete message formats over specific physical transport media. -4.1 Stream Initiate Request +5.1 Stream Initiate Request Stream Initiate Request is an addressed message sent by the Source node to the destination node to request creating the stream. This is the only method to create a stream.  The Destination Stream ID is @@ -62,124 +65,93 @@ Data content: Name Dest ID Event ID Common MTI Data Content - 6 or 12 bytes - Byte 0 Suggested Destination Stream ID, else 0xFF if - none is to be suggested. - Byte 1 Source Stream ID, value 0xFF is reserved. - Bytes 2-3 Max Buffer Size (proposed by source). + 5, 6 or 12 bytes + Byte 0-1 Max Buffer Size (proposed by source). Flags: - Stream Initiate - Request Y N 0x0CC8 Byte 4 LSB (bit 0): Deprecated, send as 0, check upon - receipt. - bit 1-7: Reserved, send as 0, check on receipt. - Additional Flags: - Byte 5 - Reserved, send as 0, ignore on receipt. - Bytes 6-11 Optional: Stream Content Type. + Byte 2-3 LSB (bit 0): Deprecated, send as 0, ignore on + Stream Initiate receipt. + Request Y N 0x0CC8 + bit 1-7: Reserved, send as 0, ignore on receipt. + Byte 4 Source Stream ID, value 0xFF is reserved. + Byte 5 Optional: Proposed Destination Stream ID, value + 0xFF is reserved. + Bytes 6-11 Optional: Stream Content UID. +       -4.2 Stream Initiate Reply +5.2 Stream Initiate Reply - Name Dest ID Event ID Common MTI Data Content - 6 bytes + optional error string - Byte 0 Destination Stream ID, value 0xFF is reserved. - Byte 1 Source Stream ID, value 0xFF is reserved. - Byte 2-3 Max Buffer Size (final value), or zero if the - request is rejected. - Error code: + Name Dest ID Event ID Common MTI Data Content + 6 bytes + optional error string + Bytes 0-1 Max Buffer Size (final value), or zero if + the request is rejected. + Flags and Error Codes: - 0x8000 for accept, no errors + 0x8000 Accept, no errors - 0x1000 Permanent error, specifically: + 0x0000 Accept, no errors - * 0x1040 streams not supported + 0x1000 Permanent error, specifically: - * 0x1020 source not permitted + * 0x1010 streams not supported - * 0x1010 unimplemented + * 0x1020 source not permitted - Stream Initiate * others reserved, do not send - Reply Y N 0x0868 -   - Bytes 4-5 - 0x2000 temporary error, specifically: + * 0x1040 unimplemented + Stream Initiate Reply Y N 0x0868 Bytes 2-3 + * others reserved, do not send - * 0x2040 unexpected, out of order, or - inconsistency in the message or frame sequence - received. +   - * 0x2020 buffer unavailable + 0x2000 temporary error, specifically: - * others reserved, do not send + * 0x2020 buffer unavailable -   + * 0x2040 unexpected, out of order, or + inconsistency in the message or frame + sequence received. - 0x0001 may be logically or'd with any of the above - error codes to indicate the incident is available - in a log (see the Logging protocol for more - information). + * others reserved, do not send + Byte 4 Source Stream ID, value 0xFF is reserved. + Byte 5 Destination Stream ID, value 0xFF is + reserved. + ‍       Bytes 6... Optional zero-terminated human readable + error string.   -4.3 Stream Data Send +5.3 Stream Data Send Name Dest ID Event ID Common MTI Data Content - 1 – Bytes + 1 to Bytes Stream Data Send Y N 0x1F88 Byte 0 Destination Stream ID, value 0xFF is reserved. Byte 1 ... Stream payload bytes.   -4.4 Stream Data Proceed +5.4 Stream Data Proceed Name Dest ID Event ID Common MTI Data Content - 2 Bytes - Stream Data Proceed Y N 0x0888 Byte 0 Destination Stream ID, value 0xFF is reserved. - Byte 1 Source Stream ID, value 0xFF is reserved. - -4.5 + 2 or 4 Bytes + Stream Data Proceed Y N 0x0888 Byte 0 Source Stream ID, value 0xFF is reserved. + Byte 1 Destination Stream ID, value 0xFF is reserved. + ‍       Byte 2-3 Optional Flags -4.6 Stream Data Control +5.5 Stream Data Complete Name Dest ID Event ID Common MTI Data Content - 2 or 6 Bytes - Byte 0 Destination Stream ID, value 0xFF is reserved. - Byte 1 Source Stream ID, value 0xFF is reserved. - Type (bits 0..3): - - 0x0:  reserved - - 0x1:  ping - - 0x2:  pong - - 0x3:  completed - - 0x4:  abort - - Stream Data Control Y N 0x08A8 Flags (bits 4..7): - Byte 2 - 0x1:  1 if this message is from the stream - source to stream destination, 0 if this - message is from the stream destination to the - stream source. - - 0x2:  1 if payload count is valid, 0 if - payload count is not valid. - - 0x4:  reserved - - 0x8:  1 if stream closed, currently only valid - for a pong type, all other types must send 0 - and ignore upon receipt. - Bytes 3-5 Number of payload bytes transferred in total - (sum of all data segments) % 224. + 2, 4 or 7 Bytes + Byte 0 Source Stream ID, value 0xFF is reserved. + Stream Data Control Y N 0x08A8 Byte 1 Destination Stream ID, value 0xFF is reserved. + Bytes 2-3 Optional Flags + Bytes 4-6 Optional number of payload bytes transferred + in total (sum of all data segments) % 224.   - 5 States (Normative) + 6 States (Normative) A stream can be in one of the following states: @@ -197,9 +169,9 @@ Open. The Destination node has sent a Stream Initiate Reply to the Source node with the Source Stream ID and the Destination Stream ID and the error code set to Accepted. - 6 Interactions (Normative) + 7 Interactions (Normative) -6.1 Opening a stream with announcement +7.1 Opening a stream with announcement In this method the streaming protocol works together with a higher-level protocol. It is the higher-level protocol that requests the transfer, and defines the format, contents and possibly the @@ -208,7 +180,7 @@ / Non-existent state. The Source node transmits the Source Stream ID to the Destination node in an addressed message of the higher-level protocol. The Source node then waits for a reception acknowledgement of the higher-level protocol (such as a Datagram Received OK) from the Destination node. - If this acknowledgement is positive, the Stream transitions to the Announced state, if it returns an + If this acknowledgement is positive, the Stream transitions to the Announced state; if it returns an error, the Stream is considered to be in the Closed state. If the Destination node can not handle the streaming protocol, it should return an error in the higher-level protocol. @@ -218,27 +190,27 @@ higher-level protocol specification requires. The Source node proposes a Max Buffer Size for the transmission. Sending this message transitions the stream to the Initiated state. - The Destination node is then required to send a Stream Initiate Reply, or one of the general error - messages defined in the Message Network Standard, such as Optional Interaction rejected, or Terminate - Due To Error. When possible, a Stream Initiate Reply should be sent instead of a general error message. + The Destination node is then required to send a Stream Initiate Reply, or the general error message + Optional Interaction Rejected as defined in the Message Network Standard. When possible, a Stream + Initiate Reply with an error code should be sent instead of a general error message. To accept the stream and transition to Open state, the Destination node must allocate a Destination Stream ID that's not currently in use with the same Source node, reply with a Stream Initiate Reply message, with the error code having the Accept bit set, the Source Stream ID set to the stream ID sent by the Source node, the Destination Stream ID filled in, and the Max Buffer Size set to a non-zero value - that is at less, than or equal to what was proposed in the Stream Initiate Request. Sending this message + that is at less than or equal to what was proposed in the Stream Initiate Request. Sending this message transitions the stream to Open state, with the Max Buffer Size as entered in the Stream Initiate Reply message. - To reject the stream, the Destination node must send a general error message, or a Stream Initiate Reply - message with the appropriate Source Stream ID, zero Max Buffer Size and the error code set to a value - with the Accept bit clear and one of the Temporary error or Permanent error bits set. When possible, a - Stream Initiate Reply should be sent instead of a general error message. When sending a general error - message, when possible, the MTI value should be set to the MTI of Stream Initiate Request. A rejected - stream is transitioned to the Closed state. If needed, the interaction may be re-started using the - higher-level protocol. + To reject the stream, the Destination node must send an Optional Interaction Rejected error message, or + a Stream Initiate Reply message with the appropriate Source Stream ID, zero Max Buffer Size and the + error code with either the Temporary error or Permanent error bits set. When possible, a Stream Initiate + Reply should be sent instead of the Optional Interaction Rejected error message. When sending an + Optional Interaction Rejected, the MTI value must be set to the MTI of Stream Initiate Request. A + rejected stream is transitioned to the Closed state. If needed, the interaction may be re-started using + the higher-level protocol. -6.2 Opening a stream without announcement +7.2 Opening a stream without announcement This method may only be used if the interaction is started by the Source node. @@ -246,12 +218,12 @@ Initiate Request to the Destination node. This message must carry a 6-byte Stream Content UID, which specifies for the destination node the content and format of the data carried, and transitions the stream to the Initiated state. From this state onwards the interaction continues as in described in - Section 6.1. + Section 7.1 . If the Stream Content UID is unknown to the Destination node, it should use the Permanent Error, Unimplemented error code in the rejection message. -6.3 Data transfer +7.3 Data transfer Once a stream transitioned to Open state, the Source node may start sending payload bytes using Stream Data Send messages. The source node may send up to Max Buffer Size bytes using one or more Stream Data @@ -260,13 +232,15 @@ When the Destination node is ready to receive another Max Buffer Size bytes, it sends a Stream Data Proceed message to the Source node. This signals to the Source node that it may send another Max Buffer Size bytes. Each Stream Data Proceed message, irrespectively of when it was sent, extends the window of - allowed payload bytes to be sent by Max Buffer Size. + allowed payload bytes to be sent by Max Buffer Size. A Stream Data Proceed may be sent before an entire + buffer of data is received, but no more than one Stream Data Proceed may be provided in advance in this + way. - If the Source node has no more data to send, but wants to keep the stream open, it may, but is not - required to periodically send Stream Data Send messages with zero payload bytes to ensure that the + If the Source node has no more data to send, but wants to keep the stream open it may, but is not + required to, periodically send Stream Data Send messages with zero payload bytes to ensure that the destination node and any gateways in the transfer path are still aware of the stream transmission. -6.4 Closing the stream +7.4 Closing the stream The Source node signals completion of the transfer by sending a Stream Data Complete message to the Destination node, providing the Source Stream ID and the Destination Stream ID with it. Optionally the @@ -276,12 +250,11 @@ The Stream Data Complete message transitions the stream to Closed state. - The Stream is closed if the Destination node sends a Terminate Due To Error or Optional Interaction - Rejected message with no MTI value or with the MTI value of Stream Data Send; or if the Source node - sends a Terminate Due To Error or Optional Interaction Rejected message with no MTI value or with the - MTI value of Stream Data Proceed. When possible, the specific MTI value should be used. + The Stream is closed if the Destination node sends a Terminate Due To Error message with the MTI value + of Stream Data Send; or if the Source node sends a Terminate Due To Error message with the MTI value of + Stream Data Proceed. -6.5 Special provisions for Gateways +7.5 Special provisions for Gateways A Gateway needs to be prepared for buffering up to Max Buffer Size bytes for each open stream. It may reduce the Max Buffer Size in the Stream Initiate Request message if the buffer size available is @@ -290,7 +263,7 @@ A Gateway may repackage the stream payload into fewer or more Stream Data Send messages than received, as the underlying network requires. A Gateway must emit a Stream Data Send message with empty payload - after its buffer is drained, if it has received such a message. + after its buffer is drained if it has received such a message. A Gateway may delay forwarding Stream Data Proceed messages until its internal buffer is drained, but it must not drop any of them. @@ -298,17 +271,17 @@ A Gateway must delay forwarding the Stream Data Complete message until its internal buffer is drained. A Gateway must drop all data in its internal buffer and all delayed messages if the Stream is terminated - with an error message of Terminate Due To Error or Optional Interaction Rejected. + with an error message of Terminate Due To Error. - 7 Adaptation to CAN Transport (Normative) + 8 Adaptation to CAN Transport (Normative) This section describes the CAN implementation of stream transport message formats. -7.1 Stream Data Send +8.1 Stream Data Send Stream Data Send messages are sent on a CAN network using the Frame Type of 7 (Stream Data), the Destination node alias and the Source node alias in the CAN header, and the Destination Stream ID in the - first data byte. The remaining data bytes can e filled with stream payload (up to 7 bytes per message): + first data byte. The remaining data bytes can be filled with stream payload (up to 7 bytes per frame): Name Dest ID Event ID Header Format Data-part Content Stream Data Send Y N 0x1Fdd,dsss Byte 0: Destination Stream ID @@ -316,7 +289,7 @@   -7.2 All Other Messages +8.2 All Other Messages All other messages are sent as described in the OpenLCB Message Network Standard. Note that the Stream Initiate Request message may need to be sent in two CAN frames if it contains a Stream Content UID. The @@ -324,204 +297,46 @@   - that support the Streaming protocol must keep track of the Stream states - - Optionally, the destination can request that the source start a transfer. This uses some mechanism not - discussed here, e.g. Datagram messages. - - The source sends an "Stream Initiate Request (Addressed)" to the destination. It carries a "Max Buffer - Size" value (2 bytes), a “Type Included” boolean flag (1 bit in a byte; see below for meaning) and a - "Source Stream ID" (1 byte) in the data section. The combination of source, destination, and Source - Stream ID must uniquely identify this stream transmission. - - If the destination node does not wish to receive the stream, it returns a "Stream Initiate Reply - (Addressed)" marked “reject”, with a "Max Buffer Size" value (2 bytes) of zero, the “Type Included” - boolean flag (1 bit in a flag byte), "Source Stream ID" (1 byte) and "Destination Stream ID" (1 byte). - The message includes flags set to represent exactly one of several conditions: - - “Permanent error” - This node does not process streams, will not accept a stream from this source, or - for some other reason will not ever be able to accept this stream. The Stream Initiate Request should - not be retransmitted. Optionally, the node can mark the reply with one or more of several conditions: - - “Information Logged” - the node supports the logging protocol and information was logged for later - retrieval. - - “Invalid Stream Request” - something made the stream request improper, such as longer than the max - permitted length. Proper requests might be acceptable. - - “Source not permitted” - streams from this source will never be accepted - - “Streams not accepted” - this node will not accept stream requests under any circumstance. A node can - also reject the interaction instead of sending Stream Initiate Reply with this code. - - “Buffer shortage, resend” - The node isn't able to receive the stream because of a shortage of buffers. - The sending node should resend at its convenience. - - “Stream rejected, out of order” - Should not happen, but an internal inconsistency was found in the CAN - frames making up a stream. Sender can try again if desired. - - If the destination node wishes to receive the stream, it returns a "Stream Initiate Reply (Addressed)" - marked “accept”, with a "Max Buffer Size" value (2 bytes), the “Type Included” boolean flag (1 bit in a - byte), "Source Stream ID" (1 byte) and "Destination Stream ID" (1 byte). 
The "Max Buffer Size" is less - than or equal to the value in the Initiate Stream Request, and is the negotiated buffer size for this - transfer. If it's zero, the request to start the stream has been rejected, and the exchange is over. The - source can try again later.
The Source Stream ID is the same as the value in the Initiate Stream - Request, and is returned for identification and the convenience of the source. The destination doesn't - do anything with it except return it. The source can use it to match up multiple operations, as a way of - identifying buffers, or for any other purpose.
The Destination Stream ID is used to tag the data sent to - the destination. It has no meaning to the source. The destination can, but need not, use it to associate - the stream data with a particular buffer or usage. Multiple simultaneous streams can use the same - Destination Stream ID value. - - The source starts sending bytes using Stream Data Send (Addressed) messages, each carrying up to the - message size limit on the particular wire protocol and the Source and Destination Stream ID. After - sending exactly Max Buffer Size bytes in one or more messages, the source pauses. - - If the “Type Included” flag was true during initialization, the first 6 bytes of the data are a unique - data-type indicator. These are allocated the same way that Node IDs, etc, are allocated, but from a - separate name space. If that “Type Included” flag is false, some higher-level protocol must identify the - data-type of the stream data. The idea is that a stream is a lot of data; there's not much use for one - otherwise because of the setup overhead (code and time). So six bytes for a stream type identifier isn't - a large cost (unlike e.g. a datagram, where it would be a 10% overhead). A UID as a Stream type ID has - the advantage that it's not ever going to collide, so people developing protocols don't have to - coordinate it (e.g. useful for future expansion, where a protocol can be locally developed and then - deployed more widely). And it still fits in the 1st CAN Frame of the stream, which simplifies what nodes - have to do to figure out what type of (unsolicited) data this is. On the other hand, streams have a - stream ID, so that a protocol can use some other mechanism (messages, datagrams, or even other streams) - to pass the information about what a specific new stream means. That means the type info at the start of - the stream isn't as necessary as in e.g. datagrams. (We really wanted to avoid situations like "The next - datagram you receive carries the data for this request", because "next" is a hard concept to ensure in - code that's doing several things independently) - - On CAN, the Stream Data Send message does not carry the Destination Stream ID field. Messages to CAN get - split into frames and forwarded without the field. Messages from CAN have to have it inserted by the - gateway as part of the translation process. The gateway can do this as part of the (optional) buffer - accumulation from frames into larger transfers. - - Upon receiving Max Buffer Size bytes via one or more messages, the destination sends a Stream Data - Proceed (Addressed) message to the source, carrying the Source Stream ID and Destination Stream ID. This - tells the source that there is enough buffer space available that another Max Buffer Size bytes can be - sent. The destination can also set flag bits (error codes) in this message to indicate that no more data - should be sent and the transmission should be terminated early. - - The destination may, but is not required to, send a Stream Data Proceed (Addressed) message to the - source before receiving the full count of Max Buffer Size bytes.  In that case, the source does not stop - transmission after sending Max Buffer Size bytes, but continues for an additional Max Buffer Size bytes - of transmission. - - When the last data has been sent, the source sends a Stream Data Complete message carrying the Source - Stream ID and Destination Stream ID to indicate that all data has been sent. Optionally, this can carry - a four-byte length of the stream data transferred for checking. A zero value, if present, means the - length is unknown. When this message is received by the destination, the transfer is complete. - - 8 Background Information - -8.1 Stream Content Type Identification - - Stream senders can, but don't have to, identify the content type (format, meaning, etc) of the data - stream at the front. As part of their private protocol, they can use whatever structure they'd like for - the data. - - General content types can be defined using unique identifiers. The “Type Included” flag identifies that - the first eight bytes of stream data are to be interpreted as stream content type information. These can - be allocated via the same mechanisms as EventIDs. Well-known ones will be published by OpenLCB. These - identifiers are recorded in a separate worksheet. - -8.2 Buffer Size - - The buffer size can be up to 216-1 bytes = 64KB-1. This is driven by the size of the initiate messages, - which we want to keep in a single CAN frame. Although some transport mechanisms might be able to - profitably use larger buffers, it seems unlikely that a layout control bus will need to have a higher - message efficiency or lower latency than this size will allow. Should that turn out to be necessary in - the future, an alternate form for the stream initialization messages can be defined for those - large-message nodes. (It's unlikely they're running over CAN) - -8.3 CAN Discussion - - On a CAN segment, the data limit is 8 bytes. This requires use of as little control information as - possible, so that as many of those eight bytes can be used for data as possible. - - OpenLCB/S9.6 CAN frames can hold the source and destination aliases in the header. A dedicated “stream - data transfer” format can also be coded entirely in the header. This then allows 8 bytes of data per - transfer if the Source Stream ID and Destination Stream ID need not be carried. By specifying that only - one stream can be transferred on a CAN link between any two nodes at a time, no Stream ID would be - needed. Although initially consider, this it thought to be too inflexible. Instead, we chose a format - where the Source Stream ID is carried, reducing the payload to seven bytes per frame. This requires that - the Source Stream ID be unique among source-destination node pairs, without relying on the Destination - Stream ID to be part of the unique stream identification. Since a node knows that it's originating the - stream, this can be ensured in advance for point-to-point connections. Gateways will need to track the - Destination Stream ID to restore it to the full format. Since they are tracking the stream in any case - for buffering purposes, this is thought to be not a large problem. - -8.4 Implementation Notes - - The "Stream Data Proceed" from the destination is clearance to send another buffer-size-worth of data. - To achieve better performance, the destination can send it before receiving the entire buffer-size-worth - of data, as soon as it has room to receive what's already been OK'd plus one more buffer size. For - example, a destination with a 4kB buffer could reply with Max Buffer Size of 2K, followed by an - immediate Stream Data Proceed, to do single overlap of the transfer. - - Intermediate nodes need to be able to handle transfers, and therefore need permission to lower the Max - Buffer Size on the outbound Stream Initiate Request message. The length in the returned Stream Initiate - Reply can't be changed, as the destination need to know when to send clearance for another buffer's - worth of data. - - A CAN ↔ Ethernet bridge might receive several kilobytes of data from the Ethernet side at a time. It's - then responsible for breaking that up into CAN frames and forwarding it. It can reduce the buffer size - of transfers if need be to ensure there's a place to store the data while this is done. - -   - Table of Contents 1 Introduction (Informative) 2 Intended Use (Informative) - 3 Definitions (Normative) - - 4  Message Formats - - 4.1 Stream Initiate Request - - 4.2 Stream Initiate Reply - - 4.3 Stream Data Send - - 4.4 Stream Data Proceed + 3 References and Context (Normative) - 4.5 + 4 Definitions (Normative) - 4.6 Stream Data Control + 5  Message Formats - 5 States (Normative) + 5.1 Stream Initiate Request - 6 Interactions (Normative) + 5.2 Stream Initiate Reply - 6.1 Opening a stream with announcement + 5.3 Stream Data Send - 6.2 Opening a stream without announcement + 5.4 Stream Data Proceed - 6.3 Data transfer + 5.5 Stream Data Complete - 6.4 Closing the stream + 6 States (Normative) - 6.5 Special provisions for Gateways + 7 Interactions (Normative) - 7 Adaptation to CAN Transport (Normative) + 7.1 Opening a stream with announcement - 7.1 Stream Data Send + 7.2 Opening a stream without announcement - 7.2 All Other Messages + 7.3 Data transfer - 8 Background Information + 7.4 Closing the stream - 8.1 Stream Content Type Identification + 7.5 Special provisions for Gateways - 8.2 Buffer Size + 8 Adaptation to CAN Transport (Normative) - 8.3 CAN Discussion + 8.1 Stream Data Send - 8.4 Implementation Notes + 8.2 All Other Messages   diff --git a/drafts/generated/StreamTransportTN.txt b/drafts/generated/StreamTransportTN.txt index 9af7a78a..d02c25a7 100644 --- a/drafts/generated/StreamTransportTN.txt +++ b/drafts/generated/StreamTransportTN.txt @@ -1,9 +1,9 @@ 1 Introduction - This Technical Note contains informative discussion and background for the corresponding “OpenLCB Stream - Transport Standard”. + This Technical Note contains informative discussion and background for the corresponding OpenLCB Stream + Transport Standard. - Some wire protocols can supoprt only very short packets/frames, e.g. CAN with a limited header and data + Some wire protocols can support only very short packets/frames, e.g. CAN with a limited header and data payload.  OpenLCB needs to be able to move large chunks of data that may not be bound by any size. A pair of nodes may be able, but are not required, to maintain up to 255 simultaneously unique @@ -42,14 +42,14 @@ * MTI allocation table. - Message Formats +   - 2.3.1 Stream Initiate Request +2.4 Definitions Stream. Other examples of streaming protocols would be TCP, or a unix pipe. The big difference to TCP is that the OpenLCB streams are uni-directional, whereas TCP connections provide bi-directional data transport. TCP also allows some out-of-band information to be sent during a live connection, although - this is a rately used feature. OpenLCB does not allow out-of-band information; any out-of-band + this is a rarely used feature. OpenLCB does not allow out-of-band information; any out-of-band information needs to be sent using different protocols. It is expected that for most streaming applications the application level protocol will define a sequence of actions, where some may use direct OpenLCB messages, others may use datagrams, and the stream @@ -60,14 +60,14 @@ a PC-based configuration tool) the sends a datagram to the server (say an OpenLCB board, or a train node), requesting a read using streams, but the server will be the Source node and the client will be the Destination node. This also means that the server will send the Stream Initiate Request message to - the client. Also the Source Stream ID will be assigned by the server, and the destination Stream ID will - be assigned by the client. + the client. The Source Stream ID will be assigned by the server, and the destination Stream ID will be + assigned by the client. Source Stream ID and Destination Stream ID, from a formal correctness perspective, are only used to disambiguate between multiple concurrently open streams between two given nodes. A stream is uniquely identified by (Source Node ID, Destination Node ID, Source Stream ID) triple, as well as the (Source Node ID, Destination Node ID, Destination Stream ID) triple. This means that Source Stream ID, by - itself, is not required to be unique; neither Destination Stream ID, b itself. + itself, is not required to be unique; neither Destination Stream ID by itself. There is a large flexibility on the implementation for how these IDs can be used. A small node may use only one fixed stream ID and not ever allow multiple streams to be open concurrently. A medium-sized @@ -77,10 +77,10 @@ is always sent along with the streaming messages so that the receiving node can perform a direct lookup of the data associated with the stream and not need to do a search for the matching state/buffer. -2.4 Message Formats +2.5 Message Formats - Every stream message type contains one or both of a Source Stream ID and/or Destination Stream ID.  An - individual stream can be identified by the combination of: + Every stream message type contains one or both of a valid Source Stream ID and/or Destination Stream ID. +  An individual stream can be identified by a combination of: 1. Source Stream ID @@ -106,29 +106,30 @@ this value for the purpose of marking a stream mapping as closed or available-for-use without requiring an additional memory location to hold this information. - 2.4.1 Stream Initiate Request + 2.5.1 Stream Initiate Request - This is an addressed message sent from the stream source node to the stream destination node.  It - contains a unique Source Stream ID, a suggested Max Buffer Size (in bytes), reserved flag bits, and - optionally a Stream Content Type. + Stream Initiate Request is an addressed message sent from the stream source node to the stream + destination node.  It contains a suggested Max Buffer Size (in bytes), reserved flag bits, a unique + Source Stream ID, an optional proposed Destination Stream ID and optionally a Stream Content Type. - The suggested Max Buffer Size chosen by the stream source is somewhat arbitrary, but could be - implementation specific.  The resulting final Max Buffer Size of the stream may be smaller, but must + The suggested Max Buffer Size chosen by the stream source is somewhat arbitrary and can be + implementation specific.  The resulting final Max Buffer Size of the stream may be smaller but must never be larger than this initially suggested value. - 2.4.2 Stream Initiate Reply + 2.5.2 Stream Initiate Reply - This is an addressed message sent from the stream destination node to the stream source node in response - to a Stream Initiate Request.  It contains a unique Destination Stream ID, a unique Source Stream ID, a - final Max Buffer Size (in bytes), and an error code field. + Stream Initiate Reply is an addressed message sent from the stream destination node to the stream source + node in response to a Stream Initiate Request.  It contains a final Max Buffer Size (in bytes), a field + for flags, a unique Source Stream ID, a unique Destination Stream ID, and in the case of an error an + optional human-readable null-terminated string describing the error. The final Max Buffer Size must be less than or equal to the suggested Max Buffer Size in the Stream Initiate Request being replied to. - 2.4.3 Stream Data Send + 2.5.3 Stream Data Send - This is an addressed message sent from the stream source node to the stream destination node.  It - contains the Destination Stream ID followed by the payload bytes. + Stream Data Send is an addressed message sent from the stream source node to the stream destination + node.  It contains the Destination Stream ID followed by the payload bytes. The stream source node must maintain an internal count that is initialized to Max Buffer Size.  For every single payload byte sent, the source must decrement this count by one.  A single Stream Data Send @@ -140,59 +141,48 @@ result in the destination node (and any gateways in between) flushing any internal buffering for efficiency that may be occurring. - 2.4.4 Stream Data Proceed - - This is an addressed message sent from the stream destination node to the stream source node.  The - destination node sends this message when it is able to receive an additional Max Buffer Size bytes of - data beyond what the stream destination node has already informed the stream source node it can receive. -  When the stream source node receives this message, it can add Max Buffer Size to its internal count of - bytes that it may send with Stream Data Send messages.  It is possible, and acceptable, that through - this mechanism, the count internal to the stream source node could increment to a value larger than Max - Buffer Size. - - 2.4.5 Stream Ping - -   - - 2.4.6 Stream Pong + 2.5.4 Stream Data Proceed -   + Stream Data Proceed is an addressed message sent from the stream destination node to the stream source + node.  The destination node sends this message when it is able to receive an additional Max Buffer Size + bytes of data beyond what the stream destination node has already informed the stream source node it can + receive.  When the stream source node receives this message, it can add Max Buffer Size to its internal + count of bytes that it may send with Stream Data Send messages.  It is possible, and acceptable, that + through this mechanism, the count internal to the stream source node could increment to a value larger + than Max Buffer Size. - 2.4.7 Stream Data Complete + 2.5.5 Stream Data Complete This is an addressed message sent from the stream source node to the stream destination node for the purpose of gracefully shutting down a stream.  This message contains a Source Stream ID, Destination - Stream ID, and optionally, a 4-byte value representing the total number of data bytes sent while the - stream was open.  Once this message is sent, the stream source node may free up the previously allocated - Source Stream ID, along with any metadata associated with the stream. + Stream ID, optional two bytes of flags, and optionally, a 4-byte value representing the total number of + data bytes sent while the stream was open.  Once this message is received, the stream destination node + may free up the previously allocated Source Stream ID, along with any metadata associated with the + stream. -2.5 States +2.6 States - A stream either exists or doesn't; those are states.  There are also some un-named intermediate states - during setup and take down. + A stream either exists or doesn't; those are states.  There are also some intermediate states during + setup and take down. -2.6 Interactions +2.7 Interactions Optionally, the destination can request that the source start a transfer. This uses some mechanism not discussed here, e.g. Datagram messages. - The source sends an "Stream Initiate Request (Addressed)" to the destination. It carries a "Max Buffer - Size" value (2 bytes), a “Type Included” boolean flag (1 bit in a byte; see below for meaning) and a - "Source Stream ID" (1 byte) in the data section. The combination of source, destination, and Source - Stream ID must uniquely identify this stream transmission. + The source sends a "Stream Initiate Request (Addressed)" to the destination. It carries a "Max Buffer + Size" value (2 bytes), and a "Source Stream ID" (1 byte) in the data section. The combination of source, + destination, and Source Stream ID must uniquely identify this stream transmission. If the destination node does not wish to receive the stream, it returns a "Stream Initiate Reply - (Addressed)" marked “reject”, with a "Max Buffer Size" value (2 bytes) of zero, the “Type Included” - boolean flag (1 bit in a flag byte), "Source Stream ID" (1 byte) and "Destination Stream ID" (1 byte). - The message includes flags set to represent exactly one of several conditions: + (Addressed)" marked “reject”, with a "Max Buffer Size" value (2 bytes) of zero, "Source Stream ID" (1 + byte) and "Destination Stream ID" (1 byte). The message includes flags set to represent exactly one of + several conditions: “Permanent error” - This node does not process streams, will not accept a stream from this source, or for some other reason will not ever be able to accept this stream. The Stream Initiate Request should not be retransmitted. Optionally, the node can mark the reply with one or more of several conditions: - “Information Logged” - the node supports the logging protocol and information was logged for later - retrieval. - “Invalid Stream Request” - something made the stream request improper, such as longer than the max permitted length. Proper requests might be acceptable. @@ -208,25 +198,25 @@ frames making up a stream. Sender can try again if desired. If the destination node wishes to receive the stream, it returns a "Stream Initiate Reply (Addressed)" - marked “accept”, with a "Max Buffer Size" value (2 bytes), the “Type Included” boolean flag (1 bit in a - byte), "Source Stream ID" (1 byte) and "Destination Stream ID" (1 byte). 
The "Max Buffer Size" is less - than or equal to the value in the Initiate Stream Request, and is the negotiated buffer size for this - transfer. If it's zero, the request to start the stream has been rejected, and the exchange is over. The - source can try again later.
The Source Stream ID is the same as the value in the Initiate Stream - Request, and is returned for identification and the convenience of the source. The destination doesn't - do anything with it except return it. The source can use it to match up multiple operations, as a way of - identifying buffers, or for any other purpose.
The Destination Stream ID is used to tag the data sent to - the destination. It has no meaning to the source. The destination can, but need not, use it to associate - the stream data with a particular buffer or usage. Multiple simultaneous streams can use the same - Destination Stream ID value. + marked “accept”, with a "Max Buffer Size" value (2 bytes), "Source Stream ID" (1 byte) and "Destination + Stream ID" (1 byte). 
The "Max Buffer Size" is less than or equal to the value in the Initiate Stream + Request, and is the negotiated buffer size for this transfer. If it's zero, the request to start the + stream has been rejected, and the exchange is over. The source can try again later.
 The Source Stream + ID is the same as the value in the Initiate Stream Request, and is returned for identification and the + convenience of the source. The destination doesn't do anything with it except return it. The source can + use it to match up multiple operations, as a way of identifying buffers, or for any other purpose.
The + Destination Stream ID is used to tag the data sent to the destination. It has no meaning to the source. + The destination can, but need not, use it to associate the stream data with a particular buffer or + usage. Multiple simultaneous streams originating  in different nodes can use the same Destination Stream + ID value. The source starts sending bytes using Stream Data Send (Addressed) messages, each carrying up to the message size limit on the particular wire protocol and the Source and Destination Stream ID. After sending exactly Max Buffer Size bytes in one or more messages, the source pauses. - If the “Type Included” flag was true during initialization, the first 6 bytes of the data are a unique + If a Stream Type was included during initialization, the first 6 bytes of the data are the same unique data-type indicator. These are allocated the same way that Node IDs, etc, are allocated, but from a - separate name space. If that “Type Included” flag is false, some higher-level protocol must identify the + separate name space. If no Stream Type was included, some higher-level protocol must identify the data-type of the stream data. The idea is that a stream is a lot of data; there's not much use for one otherwise because of the setup overhead (code and time). So six bytes for a stream type identifier isn't a large cost (unlike e.g. a datagram, where it would be a 10% overhead). A UID as a Stream type ID has @@ -269,10 +259,9 @@ stream at the front. As part of their private protocol, they can use whatever structure they'd like for the data. - General content types can be defined using unique identifiers. The “Type Included” flag identifies that - the first eight bytes of stream data are to be interpreted as stream content type information. These can - be allocated via the same mechanisms as EventIDs. Well-known ones will be published by OpenLCB. These - identifiers are recorded in a separate worksheet. + General content types can be defined using unique identifiers. These can be allocated via the same + mechanisms as EventIDs. Well-known ones will be published by OpenLCB. These identifiers are recorded in + a separate worksheet. 3.2 Buffer Size @@ -292,13 +281,13 @@ data transfer” format can also be coded entirely in the header. This then allows 8 bytes of data per transfer if the Source Stream ID and Destination Stream ID need not be carried. By specifying that only one stream can be transferred on a CAN link between any two nodes at a time, no Stream ID would be - needed. Although initially consider, this it thought to be too inflexible. Instead, we chose a format - where the Source Stream ID is carried, reducing the payload to seven bytes per frame. This requires that - the Source Stream ID be unique among source-destination node pairs, without relying on the Destination - Stream ID to be part of the unique stream identification. Since a node knows that it's originating the - stream, this can be ensured in advance for point-to-point connections. Gateways will need to track the - Destination Stream ID to restore it to the full format. Since they are tracking the stream in any case - for buffering purposes, this is thought to be not a large problem. + needed. Although initially considered, this was thought to be too inflexible. Instead, we chose a format + where the Destination Stream ID is carried, reducing the payload to seven bytes per frame. This requires + that the Destination Stream ID be unique among source-destination node pairs, without relying on the + Source Stream ID to be part of the unique stream identification. Since a node knows that it's + originating the stream, this can be ensured in advance for point-to-point connections. Gateways will + need to track the Source Stream ID to restore it to the full format. Since they are tracking the stream + in any case for buffering purposes, this is thought to be not a large problem. 3.4 Implementation Notes @@ -331,27 +320,23 @@ 2.3 References and Context - 2.3.1 Stream Initiate Request - - 2.4 Message Formats - - 2.4.1 Stream Initiate Request + 2.4 Definitions - 2.4.2 Stream Initiate Reply + 2.5 Message Formats - 2.4.3 Stream Data Send + 2.5.1 Stream Initiate Request - 2.4.4 Stream Data Proceed + 2.5.2 Stream Initiate Reply - 2.4.5 Stream Ping + 2.5.3 Stream Data Send - 2.4.6 Stream Pong + 2.5.4 Stream Data Proceed - 2.4.7 Stream Data Complete + 2.5.5 Stream Data Complete - 2.5 States + 2.6 States - 2.6 Interactions + 2.7 Interactions 3 Background Information