When processing characters, a newline is replaced by a space regardless of compile/execute mode, and then process_token is entered. When the character 0x0 is found, the token size is set to 0 and ok is called because it considers the last token to have been processed.
This procedure should be changed somehow to continue processing more tokens on the next line when in compilation mode (STATE = 1).