arrow-ipc: Extend writer benchmarks to include dictionaries

[JakeDern]

IPC writer benchmark test data do not currently include any dictionary encoded columns:

arrow-rs/arrow-ipc/benches/ipc_writer.rs

Lines 74 to 107 in cecbc72

	fn create_batch(num_rows: usize, allow_nulls: bool) -> RecordBatch {
	let schema = Arc::new(Schema::new(vec![
	Field::new("c0", DataType::Int32, true),
	Field::new("c1", DataType::Utf8, true),
	Field::new("c2", DataType::Date32, true),
	Field::new("c3", DataType::Decimal128(11, 2), true),
	]));
	let mut a = Int32Builder::new();
	let mut b = StringBuilder::new();
	let mut c = Date32Builder::new();
	let mut d = Decimal128Builder::new()
	.with_precision_and_scale(11, 2)
	.unwrap();
	for i in 0..num_rows {
	a.append_value(i as i32);
	c.append_value(i as i32);
	d.append_value((i * 1000000) as i128);
	if allow_nulls && i % 10 == 0 {
	b.append_null();
	} else {
	b.append_value(format!("this is string number {i}"));
	}
	}
	let a = a.finish();
	let b = b.finish();
	let c = c.finish();
	let d = d.finish();
	RecordBatch::try_new(
	schema.clone(),
	vec![Arc::new(a), Arc::new(b), Arc::new(c), Arc::new(d)],
	)
	.unwrap()
	}

Dictionaries have a lot of special handling in IPC writer code, which we want to optimize, so it makes sense to add a benchmark focused on these so that we can measure any perf impact.

Relevant thread from a recent PR: #10044 (comment)

CC: @alamb @Rich-T-kid

[Rich-T-kid]

@JakeDern were you planning on making a new benchmark or updating the existing benchmarks?
FWIW I think it'd be worth also isolating it into its own benchmark (writer & reader). This has to do with what you mentioned:

Dictionaries have a lot of special handling in IPC writer code, which we want to optimize.

Since there is so much other logic, it'd make sense to have benchmarks that focus on small sections, for example _encode_dictionaries(), encode_dictionaries(), and the DictionaryTracker struct.
I also think the dictionary-focused benchmarks could expand the structure of the benchmarks to cover different patterns such as the streaming behavior that the dictionary format was built around, arrow-ipc docs. It would be nice to have benchmarks that validate/check that the buffer space used to track dictionary mappings is reused instead of repeatedly allocated and destroyed.
(from the docs)

Alternatively, if isDelta is set to false, then the dictionary replaces the existing dictionary for the same ID.

like I mentioned before I haven't looked to closely at the dictionary path, but feel free to tag me in the benchmarks PR & ill be happy to take a look!

[JakeDern]

@JakeDern were you planning on making a new benchmark or updating the existing benchmarks? FWIW I think it'd be worth also isolating it into its own benchmark (writer & reader). This has to do with what you mentioned:

Dictionaries have a lot of special handling in IPC writer code, which we want to optimize.

Since there is so much other logic, it'd make sense to have benchmarks that focus on small sections, for example _encode_dictionaries(), encode_dictionaries(), and the DictionaryTracker struct. I also think the dictionary-focused benchmarks could expand the structure of the benchmarks to cover different patterns such as the streaming behavior that the dictionary format was built around, arrow-ipc docs. It would be nice to have benchmarks that validate/check that the buffer space used to track dictionary mappings is reused instead of repeatedly allocated and destroyed. (from the docs)

Alternatively, if isDelta is set to false, then the dictionary replaces the existing dictionary for the same ID.

like I mentioned before I haven't looked to closely at the dictionary path, but feel free > to tag me in the benchmarks PR & ill be happy to take a look!

I think we're on the same page! I added new benchmarks for the writer side with different focuses:

• StreamWriter dictionary messages
• StreamWriter delta dictionary messages
• FileWriter delta dictionary messages

The delta and non-delta paths definitely exercise different paths and have different perf characteristics. I didn't go quite to the level of granularity of functions like encode_dictionary or the DictionaryTracker struct, but my opinion is as long as any perf improvements/regressions are detectable in the current set of benchmarks then the coverage is ok.

The topic of making sure that dictionary mappings are re-used instead of repeatedly allocated and destroyed is sadly tricky, but something that I ensured the benchmarks. I think we can simultaneously (1) more reliably emit delta dictionaries and (2) make the emission of delta dictionaries more efficient, but IMO both are more of an API level problem that I tried to describe in #8134.

On the subject of API issues for the writer - I think IpcDataGenerator should also be a future target which has public apis like encode which is tied to the old EncodedData pattern and likely shouldn't be used. There are probably other APIs worth deprecating in the module too:

arrow-rs/arrow-ipc/src/writer.rs

Lines 546 to 553 in 826b808

	pub fn encode(
	&self,
	batch: &RecordBatch,
	dictionary_tracker: &mut DictionaryTracker,
	write_options: &IpcWriteOptions,
	compression_context: &mut CompressionContext,
	) -> Result<(Vec<EncodedData>, EncodedData), ArrowError> {
	let encoded_dictionaries = self.encode_all_dicts(

.

[Rich-T-kid]

nice! i left some non-blocking comments.