End delimiter confusion when it appears in the payload

[SvenNouwens]

The title says it all, the unit16 end delimiter occasionally appears in by payload containing, amongst others, sensor data. As expected, the misidentified packet gets dropped as the CRC doesn't match.

As there is no way to guarantee the end delimiter never appears in my payload, there is always a possiblity that packets gets dropped for this reason. I was wonder if I can specify some fixed packet size to circumvent this? Or does Serial Studio provide other methods for dealing with this issue?

Currently I'm just going to use an uint32 end delimiter to push the problem further down the road, but I feel that this is not an elegant solution for the problem at hand.

Any advice is welcome!

[alex-spataru]

Hi!

Could you please provide an example of the raw data you send to Serial Studio and your project file?

I will proceed to testing and fixing the issue on the evening. If possible, I will also add an option to specify the number of bytes to receive, this could solve a lot of issues in large frames which don’t have any delimiters.

Thanks,
Alex Spataru

[SvenNouwens]

Hi, thanks for the answer. The data I'm sending essentially looks like:

struct PayloadData {
  uint16_t start_delimiter;  // 0xABCD
  uint8_t packet_counter;   // Packet counter to track dropped packets
  uint16_t data [10];             // Telemetry payload
  uint16_t end_delimiter;    // 0x1234
  uint16_t crc16;            
} __attribute__((packed));    // Ensure no compiler padding between bytes

union PayloadUnion {
  PayloadData data;
  byte binary[sizeof(PayloadData)];
};

I'm subsequently sending PayloadUnion.binary

The project file is nothing fancy. Essentially extended the HEX ADC example to include an end delimiter.

An option to set the expected number of bytes would by great!

[alex-spataru]

Why This Happens

The frame reader uses the Knuth-Morris-Pratt (KMP) pattern matching algorithm to search for delimiter sequences in the incoming byte stream. The algorithm finds the first occurrence of your delimiter pattern. However, it has no way to distinguish between:

• Delimiter bytes that are part of your sensor data payload
• The actual end-of-frame delimiter

When your uint16 delimiter appears in the payload data, here's what happens:

Actual frame structure:

• [SENSOR_DATA_1][0xABCD in payload][SENSOR_DATA_2][0xABCD delimiter][CRC]

What FrameReader sees:

• Frame 1: [SENSOR_DATA_1] ← CRC validation fails, frame discarded
• Remaining buffer: [0xABCD][SENSOR_DATA_2][0xABCD][CRC] ← malformed data

The checksum validation correctly detects the problem (which is why the packet gets dropped), but by that point the frame boundaries have already been parsed incorrectly.

Possible Solution: Use No Delimiters Mode with Fixed-Length Frames

Serial Studio supports a frame detection mode that bypasses delimiter scanning entirely and passes raw data directly to your JavaScript parser. You can then implement your own framing logic, such as a fixed length buffer parsing:

[START_SEQUENCE][LENGTH][PAYLOAD...][CHECKSUM]

JavaScript parser example with buffering:

/**
 * Fixed-Length Frame Parser with End Delimiter
 *
 * Solves the "delimiter in payload" problem by using fixed frame length.
 * User only needs to configure END_SEQUENCE and FRAME_LENGTH.
 *
 * HOW IT WORKS:
 * 1. Searches for END_SEQUENCE in buffer
 * 2. Extracts exactly FRAME_LENGTH bytes BEFORE the delimiter
 * 3. Delimiter bytes in payload are ignored (only the last one matters)
 */

//------------------------------------------------------------------------------
// USER CONFIGURATION
//------------------------------------------------------------------------------

// Define your end delimiter (as array of bytes)
var END_SEQUENCE = [0x0D, 0x0A]; 

// Define your fixed frame length in bytes (excluding delimiter)
var FRAME_LENGTH = 10;  

//------------------------------------------------------------------------------
// Global buffer
//------------------------------------------------------------------------------

var buffer = [];

//------------------------------------------------------------------------------
// Frame Parser
//------------------------------------------------------------------------------

function parse(frame) {
  // Append new data to buffer
  for (var i = 0; i < frame.length; i++) {
    buffer.push(frame[i]);
  }

  // Search for end delimiter
  while (buffer.length >= FRAME_LENGTH + END_SEQUENCE.length) {
    var delimiterIndex = findDelimiter();

    // No delimiter found, keep last (FRAME_LENGTH + END_SEQUENCE.length - 1) bytes
    if (delimiterIndex === -1) {
      if (buffer.length > FRAME_LENGTH + END_SEQUENCE.length) {
        buffer.splice(0, buffer.length - (FRAME_LENGTH + END_SEQUENCE.length - 1));
      }
      break;
    }

    // Check if we have enough data before delimiter
    if (delimiterIndex < FRAME_LENGTH) {
      buffer.splice(0, delimiterIndex + END_SEQUENCE.length);
      continue;
    }

    // Extract frame of exactly FRAME_LENGTH bytes before delimiter
    var frameStart = delimiterIndex - FRAME_LENGTH;
    var payload = buffer.slice(frameStart, delimiterIndex);

    // Remove processed frame + delimiter from buffer
    buffer.splice(0, delimiterIndex + END_SEQUENCE.length);

    // Parse and return the payload
    return parsePayload(payload);
  }

  // No complete frame yet
  return [];  
}

function findDelimiter() {
  for (var i = 0; i <= buffer.length - END_SEQUENCE.length; i++) {
    var match = true;
    for (var j = 0; j < END_SEQUENCE.length; j++) {
      if (buffer[i + j] !== END_SEQUENCE[j]) {
        match = false;
        break;
      }
    }
    if (match) {
      return i;
    }
  }
  return -1;
}

//------------------------------------------------------------------------------
// Payload Parser (MODIFY THIS IF NEEDED)
//------------------------------------------------------------------------------

/** 
 * Returns array of byte values as strings
 * Modify this function if you need to parse your data differently
 * (e.g., combine bytes into 16-bit values, apply scaling, etc.)
 */
function parsePayload(payload) {
  var result = [];
  for (var i = 0; i < payload.length; i++) {
    result.push(payload[i].toString());
  }
  
  return result;
}

This approach guarantees that delimiter bytes in your payload will never cause frame loss, because you're reading an exact byte count from the length field.

Future Enhancements

I'll consider adding the following features to make this easier:

1. Built-in COBS (Consistent Overhead Byte Stuffing) support.
2. Fixed-length frame option in the Project Editor.
3. Length-prefix protocol template.

These would eliminate the need for manual JavaScript buffering in most cases.

---

TL;DR: Switch to No Delimiters mode and implement length-based framing in JavaScript. This completely avoids the delimiter confusion problem. As you said, using a uint32 delimiter is not a proper fix, you need a protocol that's delimiter-agnostic (length-prefix, COBS, or similar).

Please let me know if you need help setting this up!