-
-
Notifications
You must be signed in to change notification settings - Fork 286
scala_export macro #1790
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
vinnybod
wants to merge
17
commits into
bazel-contrib:master
Choose a base branch
from
confluentinc:vinnybod/upstream-scala-export
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
scala_export macro #1790
Changes from all commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
28de47d
Copy scala_export from rules_jvm_external
vinnybod 3ca6acf
update with new attributes
vinnybod 7cabce5
use master commit on bazel-contrib/rules_jvm_external
vinnybod 4673fc3
bump rje to 6.9
vinnybod f8217dc
Merge branch 'master' into vinnybod/upstream-scala-export
vinnybod cda0546
remove todo
vinnybod c8fddd6
Merge branch 'master' into vinnybod/upstream-scala-export
vinnybod 6839957
add workspace dep
vinnybod 43091b5
add script for updating lockfiles
vinnybod 30f6b47
fix twitter_scrooge test
vinnybod fc90535
update test_reproducibility.sh
vinnybod ed0d468
swap diff_test to aspect due to windows error
vinnybod 0108894
add proto override
vinnybod b240682
address easy feedback:
vinnybod 3df8505
Update lockfiles
vinnybod d44d780
publish scaladoc as javadoc jar
vinnybod a60e21a
Merge branch 'master' into vinnybod/upstream-scala-export
vinnybod File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,125 @@ | ||
| #!/usr/bin/env bash | ||
| # | ||
| # Script to analyze non-determinism in -docs.jar files generated by maven_export | ||
| # Run from the rules_scala root directory | ||
| # | ||
|
|
||
| set -e | ||
|
|
||
| TARGET="//test/scala_export:external_dep-docs" | ||
| JAR_PATH="bazel-bin/test/scala_export/external_dep-docs.jar" | ||
|
|
||
| echo "=== Analyzing non-determinism in $TARGET ===" | ||
| echo "" | ||
|
|
||
| # Create temp directories for extraction | ||
| DIR1=$(mktemp -d -t docs_jar_build1-XXXXXXXXXX) | ||
| DIR2=$(mktemp -d -t docs_jar_build2-XXXXXXXXXX) | ||
| CACHE1=$(mktemp -d -t cache1-XXXXXXXXXX) | ||
| CACHE2=$(mktemp -d -t cache2-XXXXXXXXXX) | ||
|
|
||
| cleanup() { | ||
| rm -rf "$DIR1" "$DIR2" "$CACHE1" "$CACHE2" | ||
| } | ||
| trap cleanup EXIT | ||
|
|
||
| echo ">>> Build 1" | ||
| bazel clean 2>/dev/null | ||
| bazel build --disk_cache="$CACHE1" "$TARGET" 2>/dev/null | ||
| cat "$JAR_PATH" > "$DIR1/docs.jar" | ||
| (cd "$DIR1" && unzip -o -q docs.jar && rm docs.jar) | ||
|
|
||
| echo ">>> Build 2 (after clean and sleep)" | ||
| bazel clean 2>/dev/null | ||
| sleep 5 # Ensure different timestamps | ||
| bazel build --disk_cache="$CACHE2" "$TARGET" 2>/dev/null | ||
| cat "$JAR_PATH" > "$DIR2/docs.jar" | ||
| (cd "$DIR2" && unzip -o -q docs.jar && rm docs.jar) | ||
|
|
||
| echo "" | ||
| echo "=== Comparing jar contents ===" | ||
| echo "" | ||
|
|
||
| # Compare file lists | ||
| echo "--- Files only in build 1:" | ||
| diff <(cd "$DIR1" && find . -type f | sort) <(cd "$DIR2" && find . -type f | sort) | grep "^<" || echo "(none)" | ||
|
|
||
| echo "" | ||
| echo "--- Files only in build 2:" | ||
| diff <(cd "$DIR1" && find . -type f | sort) <(cd "$DIR2" && find . -type f | sort) | grep "^>" || echo "(none)" | ||
|
|
||
| echo "" | ||
| echo "--- Files with different content:" | ||
| DIFF_FILES=() | ||
| while IFS= read -r file; do | ||
| if [ -f "$DIR1/$file" ] && [ -f "$DIR2/$file" ]; then | ||
| if ! cmp -s "$DIR1/$file" "$DIR2/$file"; then | ||
| DIFF_FILES+=("$file") | ||
| echo " $file" | ||
| fi | ||
| fi | ||
| done < <(cd "$DIR1" && find . -type f | sort) | ||
|
|
||
| echo "" | ||
| echo "=== Showing differences in changed files ===" | ||
| echo "" | ||
|
|
||
| for file in "${DIFF_FILES[@]}"; do | ||
| echo ">>> Diff for: $file" | ||
| echo "-------------------------------------------" | ||
|
|
||
| if [[ "$file" == *.zip ]]; then | ||
| # These are zip files - extract and compare | ||
| ZIPDIR1=$(mktemp -d) | ||
| ZIPDIR2=$(mktemp -d) | ||
|
|
||
| # Extract zip contents | ||
| unzip -o -q "$DIR1/$file" -d "$ZIPDIR1" 2>/dev/null || true | ||
| unzip -o -q "$DIR2/$file" -d "$ZIPDIR2" 2>/dev/null || true | ||
|
|
||
| # List what we extracted | ||
| echo " Extracted files from zip:" | ||
| (cd "$ZIPDIR1" && find . -type f) || true | ||
|
|
||
| # Compare extracted contents | ||
| found_diff=false | ||
| for zfile in $(cd "$ZIPDIR1" && find . -type f 2>/dev/null); do | ||
| if [ -f "$ZIPDIR1/$zfile" ] && [ -f "$ZIPDIR2/$zfile" ]; then | ||
| if ! cmp -s "$ZIPDIR1/$zfile" "$ZIPDIR2/$zfile"; then | ||
| found_diff=true | ||
| echo "" | ||
| echo " Difference in: $zfile" | ||
| diff "$ZIPDIR1/$zfile" "$ZIPDIR2/$zfile" | head -30 || true | ||
| fi | ||
| fi | ||
| done | ||
|
|
||
| if [ "$found_diff" = false ]; then | ||
| # Maybe the zip metadata differs, not content | ||
| echo " (zip file metadata differs, not content - showing hexdump diff)" | ||
| diff <(xxd "$DIR1/$file" | head -20) <(xxd "$DIR2/$file" | head -20) || true | ||
| fi | ||
|
|
||
| rm -rf "$ZIPDIR1" "$ZIPDIR2" | ||
| elif [[ "$file" == *.html ]] || [[ "$file" == *.js ]] || [[ "$file" == *.json ]] || [[ "$file" == *.css ]]; then | ||
| # For text files, show the actual diff | ||
| diff "$DIR1/$file" "$DIR2/$file" | head -50 || true | ||
| else | ||
| # For other binary files, show hexdump diff | ||
| echo "(binary file - showing hexdump diff)" | ||
| diff <(xxd "$DIR1/$file" | head -20) <(xxd "$DIR2/$file" | head -20) || true | ||
| fi | ||
| echo "" | ||
| done | ||
|
|
||
| echo "=== Summary ===" | ||
| echo "Total files with differences: ${#DIFF_FILES[@]}" | ||
| echo "" | ||
| echo "The hexdump shows differences in ZIP file headers at bytes 10-11 (e.g., b758 vs c758)." | ||
| echo "These bytes represent the 'last modification time' in DOS format." | ||
| echo "" | ||
| echo "The actual content (JSON files) inside the ZIPs is identical, but javadoc embeds" | ||
| echo "the current system time as file modification timestamps in the ZIP headers." | ||
| echo "This makes the -docs.jar non-deterministic even though documentation content is the same." | ||
| echo "" | ||
| echo "This is controlled by rules_jvm_external's maven_export macro, not rules_scala." |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.