Skip to content

Commit 07a61d6

Browse files
authored
Merge pull request #5 from beyonddream/add-toc-theme
rephrase few sentences and minor update to code
2 parents e9fe6f0 + 7ba6368 commit 07a61d6

1 file changed

Lines changed: 5 additions & 4 deletions

File tree

posts/post-1.md

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -92,10 +92,10 @@ object PartitionedReaderWriter {
9292

9393
## LLM Version
9494

95-
Below is the version (keeping only the relevant bits) generated by claude 3.5 sonnet.
96-
I haven't run it to verify it works but looking at the code I think it is correct - atleast in spirit :)
95+
Below is the version (keeping only the relevant bits) generated by claude 3.5 sonnet. Granted, this code is not what I would
96+
have come up with by default but the code works and it is not terribly hard to understand.
9797

98-
The scary and interesting part is, it provided me a solution in few seconds that I had only arrived at after a few days of much toil and head scratching. On the positive side, toiling and hard work provided me an increased sense of confidence in my technical ability and deepening of knowledge in Spark.
98+
The scary (or interesting) part is, it provided me a solution in few seconds that I had only arrived at after a few days of much toil and head scratching. On the positive side, toiling and hard work provided me an increased sense of confidence in my technical ability and deepening of knowledge in Spark.
9999

100100
I can only hope one can take the best of both worlds in this brave new world!
101101

@@ -118,7 +118,8 @@ def main(args: Array[String]) {
118118

119119
// Add partition columns
120120
val writeDf = sourceDf_with_paths
121-
.withColumn("partitions", extractPartitionsUDF(lit(sourceBasePath), col("source_path"))) // source_path is input_file_name()
121+
.withColumn("source_path", input_file_name())
122+
.withColumn("partitions", extractPartitionsUDF(lit(sourceBasePath), col("source_path")))
122123
.withColumn("year", col("partitions.year"))
123124
.withColumn("month", col("partitions.month"))
124125
.withColumn("day", col("partitions.day"))

0 commit comments

Comments
 (0)