When there is lot of enqueue , it fails

[abhit011]

I have been using queue like this

parallelQueue::ConcurrentQueue<myDS,mYTraits> *pMPSC_consume;

pMPSC_consume = new ConcurrentQueue(PARALLEL_QUEUE_MAX_NODES, 0, enteredNoOfThreads);

and when i call pMPSC_consume->try_enqueue() , its failing at times ....

I have given PARALLEL_QUEUE_MAX_NODES and even at times there are close to like max 10K to 20K in loop, but even then the try_enqueue fails , please let me know what i need to do to handle this

thanks

[abhit011]

typedef struct mYTraits: public parallelQueue::ConcurrentQueueDefaultTraits
{
static const size_t BLOCK_SIZE = 128;
static const size_t INITIAL_IMPLICIT_PRODUCER_HASH_SIZE = 128;
static const size_t IMPLICIT_INITIAL_INDEX_SIZE = 128;
};

and these are the Index sizes

[abhit011]

I am just producing from single thread and consuming this queue from single thread, without re-allocating further memory

[abhit011]

as per the code / Computes the correct amount of pre-allocated blocks for you based
// on the minimum number of elements you want available at any given
// time, and the maximum concurrent number of each type of producer.

Then isnt a producer thread should have 100K nodes ? but its failing when it touched 16K in a loop

[abhit011]

the the enteredNoOfThread is just 1

pMPSC_consume = new ConcurrentQueue(PARALLEL_QUEUE_MAX_NODES, 0, enteredNoOfThreads);

[abhit011]

@cameron314 , can you please help or share some details ...

[cameron314]

As documented, with an IMPLICIT_INITIAL_INDEX_SIZE of 128 and a block size of 128, that gives a max of 16k elements per implicit producer. Increase IMPLICIT_INITIAL_INDEX_SIZE or use enqueue instead of try_enqueue. The arguments given to the constructor allocate enough blocks for more elements than that in advance, but the index size is your limitation here.

[abhit011]

ohh ConcurrentQueue(PARALLEL_QUEUE_MAX_NODES, 0, enteredNoOfThreads), the no of thread are like no parallel producers, which means that each thread will be having Max 128 x 128 thats total of approx 16K nodes as per of size myDS struct,
But in this case i was creating queue of like its producing from single thread only , hence

i did something like BLOCK_SIZE = 1024, and IMPLICIT_INITIAL_INDEX_SIZE = 2048

[cameron314]

Does this resolve your issue?

[abhit011]

Yes , this i had made the change last sat . I am observing more ...
Do you have any scribbling or any doc that visualizes the parallel queue, i get that its parallel queues as per no of producers, and each producer has array of nodes to access ....

[cameron314]

Not more than what's in the comments, README, and blog post. Yes, each producer is mapped to a sub-queue which has an index or list of blocks of elements.