Skip to content

native_datafusion reports twice more files and bytes scanned #3791

@comphead

Description

@comphead

Describe the bug

spark.range(10000000).repartition(20).write.parquet(location) 
spark.read.parquet("location").show(false)
Image Image

which are twice more than original. The issue seems to be for metric only, the actual file read only once

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't workinggood first issueGood for newcomersnative_datafusionSpecific to native_datafusion scan typepriority:mediumFunctional bugs, performance regressions, broken features

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions