Restore phandles from binary representations by ukleinek · Pull Request #151 · dgibson/dtc

ukleinek · 2024-10-21T07:45:40Z

device trees generated with options -@ and -L contain information about the label names used originally and which values are phandle values. This information can be reused when compiling back to dts format to make the result much more human readable.

Also dtdiff is adapted to make use of this to generate smaller diffs without hunks e.g. for a different phandle allocation.

This superseeds pull request #93.

ukleinek · 2024-10-21T08:02:34Z

A possible followup improvement would be to restore phandles from __fixup__ .

ukleinek · 2024-10-24T06:11:09Z

Just a quick headsup: I found a dtbo file where my code crashes with -@ -L. So don't merge yet please.

ukleinek · 2024-10-24T06:29:09Z

OK, I think that's unrelated to my changes and just presents a situation to the decompiler that didn't happen before. Here is a reproducer:

$ git diff
diff --git a/treesource.c b/treesource.c
index 067647b60a17..fa1a48579489 100644
--- a/treesource.c
+++ b/treesource.c
@@ -104,6 +104,12 @@ static void write_propval_string(FILE *f, const char *s, size_t len)
 static void write_propval_int(FILE *f, const char *p, size_t len, size_t width)
 {
        const char *end = p + len;
+
+       if (len % width) {
+               fprintf(stderr, "Huh, len = %zu, width = %zu\n", len, width);
+               exit(1);
+       }
+
        assert(len % width == 0);
 
        for (; p < end; p += width) {

uwe@taurus:~/dts-decompile-labels$ cat test3.dts
/dts-v1/;

/ {
	tralala = <&somelabel>, "text making the next label unaligned",
		<&somelabel>, "some more text";

	somelabel: somenode {

	};
};
$ dtc -@ -L -I dts -O dtb test3.dts > test3.dtb
$ dtc -@ -L -I dtb -O dts test3.dtb 
/dts-v1/;

/ {
Huh, len = 37, width = 4
	tralala = <&somelabel

So I think some heuristic considers the property "tralala" an array of ints but that doesn't handle that it ends at offset 41 which is unaligned.

dgibson

I think this is a great idea. The implementation looks mostly sound but there are a few minor issues that need addressing.

dtc.c

livetree.c

dtc.c

livetree.c

treesource.c

dtdiff

dgibson · 2024-10-28T09:22:59Z

A possible followup improvement would be to restore phandles from __fixup__ .

I think you do need to implement this. For one thing processing but __local_fixups__ but not __fixups__ would just be surprising behaviour to a user, I think. Secondly, it could cause some bogus errors in some cases (see detailed comments).

ukleinek · 2024-10-28T09:29:26Z

Do you have an idea for the unaligned phandle problem?

dgibson · 2024-10-28T09:43:17Z

Do you have an idea for the unaligned phandle problem?

Sorry, I'm not sure exactly what problem you mean.

ukleinek · 2024-10-28T14:08:06Z

Do you have an idea for the unaligned phandle problem?

Sorry, I'm not sure exactly what problem you mean.

I mean the crash reported in #151 (comment) with the follow up in the next comment.

dgibson · 2024-10-30T02:07:48Z

Do you have an idea for the unaligned phandle problem?

Sorry, I'm not sure exactly what problem you mean.

I mean the crash reported in #151 (comment) with the follow up in the next comment.

Ah, right, sorry.

I think the basic problem is that previously the only case where we'd have phandle markers during decompile is with -I dts -O dts, in which case we'd also have type markers helping us figure out how to print each part of each property. The new code is putting in phandle markers, but without type markers. We should handle that, of course, but looks like there are some edge cases.

More specifically, it looks like write_propval() calls guess_value_type() to figure out a missing type only once for the whole property. Having a phandle marker does tell us that those specific 4 bytes should be in integer < ... > context. But we'll need to separately guess the type for each chunk between phandles. I suspect that might be irritatingly fiddly, but I'm afraid it's going to be a prerequisite for implementing this.

ukleinek · 2025-01-12T01:24:39Z

This addresses the crash mentioned in #151 (comment) and a few suggestions by @dgibson (which I marked as resolved). Still missing is restoring of phandles from __fixups__.

treesource.c

dtc.c

ukleinek · 2025-01-14T11:24:02Z

OK, this got bigger now than I anticipated and also contains a few fixes, but I'm happy that I did it now and I can get it out of my head and happily use it.

Quick demo:

$ cat test.dts
/dts-v1/;
/plugin/;

/ {
	tralal = <&somelabel>, "text making the",
		<&somelabel>, "some more text";

	tralala = <&somelabel>, "text making the next label unaligned",
		<&somelabel>, "some more text";

	somelabel: somenode {
		property = <&nonexisting>;
	};
};
$ dtc -@ test.dts > test.dtb

Decompiling this dtb with dtc 1.7.2 gives you:

$ /usr/bin/dtc test.dtb
/dts-v1/;

/ {
	tralal = [00 00 00 01 74 65 78 74 20 6d 61 6b 69 6e 67 20 74 68 65 00 00 00 00 01 73 6f 6d 65 20 6d 6f 72 65 20 74 65 78 74 00];
	tralala = <0x01 0x74657874 0x206d616b 0x696e6720 0x74686520 0x6e657874 0x206c6162 0x656c2075 0x6e616c69 0x676e6564 0x00 0x1736f6d 0x65206d6f 0x72652074 0x65787400>;

	somenode {
		property = <0xffffffff>;
		phandle = <0x01>;
	};

	__symbols__ {
		somelabel = "/somenode";
	};

	__fixups__ {
		nonexisting = "/somenode:property:0";
	};

	__local_fixups__ {
		tralal = <0x00 0x14>;
		tralala = <0x00 0x29>;
	};
};

With the changes from this PR applied it gets:

$ dtc test.dtb
/dts-v1/;
/plugin/;

/ {
	tralal = <&somelabel>, "text making the", <&somelabel>, "some more text";
	tralala = <&somelabel>, "text making the next label unaligned", <&somelabel>, "some more text";

	somelabel: somenode {
		property = <&nonexisting>;
		phandle = <0x01>;
	};

	__symbols__ {
		somelabel = "/somenode";
	};

	__fixups__ {
		nonexisting = "/somenode:property:0";
	};

	__local_fixups__ {
		tralal = <0x00 0x14>;
		tralala = <0x00 0x29>;
	};
};

and if test.dtb was compiled without -@ it uses &{/somenode} instead of &somelabel.

ukleinek · 2025-01-15T09:17:13Z

BTW, I didn't implement stripping out the __fixups__, __local_fixups__ and __symbols__ nodes. Mostly because it doesn't seem to be right for -I dts -O dts compilations. Instead the two fixup nodes are removed before they are generated to make sure that the information isn't duplicated.

dgibson

I've applied the first three patches, because they're sensible fixes independent of the rest of the series. I've made some comments on the next few, but I haven't completed a full review yet.

treesource.c

tests/over-determined.dts

treesource.c

ukleinek · 2025-01-25T15:25:43Z

I've applied the first three patches, because they're sensible fixes independent of the rest of the series. I've made some comments on the next few, but I haven't completed a full review yet.

The topmost commit is also simple and orthogonal to the rest of the series. That one might be eligible for fast tracking, too.
I'll look into your other feedback later.

ukleinek · 2025-07-01T10:19:47Z

The topmost commit is also simple and orthogonal to the rest of the series. That one might be eligible for fast tracking, too. I'll look into your other feedback later.

I split this into a separate PR, see #168.

This fixes `dtc -I dts -O dts` to make the file a plugin if the source file is one. Signed-off-by: Uwe Kleine-König <u.kleine-koenig@baylibre.com>

ukleinek · 2025-07-01T11:06:06Z

I think I addressed all feedback. The only open discussion is if __fixups__, __local_fixups__ and __symbols__ should be dropped and under which conditions.

dgibson · 2025-07-02T05:45:41Z

The first two patches looked fine and I was going to apply.. but the second one breaks a test case:

dtc -I dtb -O dts -o overlay_overlay_decompile.test.dts overlay_overlay.test.dtb:	PASS
dtc -I dts -O dtb -o overlay_overlay_decompile.test.dtb overlay_overlay_decompile.test.dts:PASS
dtbs_equal_ordered overlay_overlay.test.dtb overlay_overlay_decompile.test.dtb:	Starting testcase "./dtbs_equal_ordered", pid 98779
FAIL	Tag mismatch (1 != 2) at (672, 672)

It looks like the decompiled version is entirely missing the __fixups__ and __local_fixups__ which should be there. I'm guessing this eventually gets fixed by the later patches which will reconstruct the necessary source level references to remake the fixups. However, we don't like to break bisection like that, so that patch might have to be moved later to where it won't break things.

treesource.c

dgibson · 2025-07-02T05:54:02Z

treesource.c

+
+	if (*mi && (*mi)->offset == offset && is_type_marker((*mi)->type)) {
+		if (is_type_marker(type))
+			return mi;


This avoids adding the marker if it would violate the "at most one type marker per offset" invariant. Which is fine, I guess, but you're essentially failing silently, which seems like it might lead to hard to debug problems.

A helpful warning message is hard here because we don't know the affected property here. So would you prefer a warning message that is hard to understand à la: "Duplicate type marker at offset %u, type %d vs. type %d" offset, type, (*mi)->type? Or should I add a reference to the property to add_marker just for the purpose of the warning?

dgibson · 2025-07-02T05:55:46Z

treesource.c

+	}
+
+	if (*mi && (*mi)->offset == offset && type == (*mi)->type)
+		return mi;


Similar silent failure here, but this one is worse. It's valid to have several path references in a row, which will have the same offset (because they're zero-length before fixup). This logic will prevent you from adding a marker in that case.

I don't understand that. A marker never has a length (see the definition of struct marker) and I don't see how there could be more than one REF_PATH marker at a single offset.

Sorry, I was a bit unclear. Consider:

prop = &foo, &bar;

So we have two path references in a row. The first one will obviously have offset 0. But, before we actually expand &foo in fixup_path_references() we won't generate any data for the reference. That means the second reference will also have offset 0

treesource.c

livetree.c

dgibson · 2025-07-02T06:28:49Z

treesource.c

 	return &nm->next;
 }

+void property_add_marker(struct property *prop,


This belongs in livetree.c

Is this true with add_marker in treesource.c?

Both should be in livetree.c. They're helpers manipulating the "live" (pointer based, in-memory) version of the tree data structure.

tests/run_tests.sh

ukleinek · 2025-07-02T09:06:04Z

The first two patches looked fine and I was going to apply.. but the second one breaks a test case:
dtc -I dtb -O dts -o overlay_overlay_decompile.test.dts overlay_overlay.test.dtb:	PASS
dtc -I dts -O dtb -o overlay_overlay_decompile.test.dtb overlay_overlay_decompile.test.dts:PASS
dtbs_equal_ordered overlay_overlay.test.dtb overlay_overlay_decompile.test.dtb:	Starting testcase "./dtbs_equal_ordered", pid 98779
FAIL	Tag mismatch (1 != 2) at (672, 672)
It looks like the decompiled version is entirely missing the __fixups__ and __local_fixups__ which should be there. I'm guessing this eventually gets fixed by the later patches which will reconstruct the necessary source level references to remake the fixups. However, we don't like to break bisection like that, so that patch might have to be moved later to where it won't break things.

This commit uncovers a problem that we already have since commit 915daad ("Start with empty local_fixups and fixups nodes"):

$ cat test.dts 
/dts-v1/;
/plugin/;

/ {
	node {
		property = <0xffffffff>, "string";
	};
	
	__fixups__ {
		somenode = "/node:property:0";
	};
};

$ dtc -I dts -O dts test.dts 
/dts-v1/;

/ {

	node {
		property = <0xffffffff>, "string";
	};
};

I'll think about a fix.

ukleinek · 2025-07-02T09:26:11Z

I'll think about a fix.

The fix is to fold the info contained in __fixes__ (and __local_fixups__) into markers, so the commits in this PR fix and don't only cover the issue again.

ukleinek · 2025-07-02T09:50:59Z

I'll think about a fix.

The fix is to fold the info contained in __fixes__ (and __local_fixups__) into markers, so the commits in this PR fix and don't only cover the issue again.

I fixed that by sorting the respective commits before this change and expanded their commit logs to describe this detail.

Typically the info contained in __local_fixups__ and __fixups__ is autogenerated from phandles used in the tree. However before commit 915daad ("Start with empty __local_fixups__ and __fixups__ nodes") compiling a dts that has both the __local_fixups__ and/or __fixups__ and phandles that result in entries in these nodes, the fixups were duplicated: $ cat test.dts /dts-v1/; /plugin/; / { important: node { property = <&somenode>, "string"; phandle = <0x01>; self = <&important>; }; __fixups__ { somenode = "/node:property:0"; }; __local_fixups__ { node { self = <0x00>; }; }; __symbols__ { important = "/node"; }; }; $ dtc -v Version: DTC 1.7.2 $ dtc -O dts test2.dts /dts-v1/; / { important: node { property = <&somenode>, "string"; phandle = <0x01>; self = <&important>; }; __fixups__ { somenode = "/node:property:0", "/node:property:0"; }; __local_fixups__ { node { self = <0x00>, <0x00>; }; }; __symbols__ { important = "/node"; }; }; Add a test that ensures this issue isn't reintroduced later. Signed-off-by: Uwe Kleine-König <u.kleine-koenig@baylibre.com>

The add_marker() function is used to create a new marker and add it at the right spot to the relevant marker list. Use it in the add_string_markers() helper (which gets slightly quicker by it). Signed-off-by: Uwe Kleine-König <u.kleine-koenig@baylibre.com>

In the presence of (non-type) markers guess the type of each chunk between markers individually instead of only once for the whole property. Note that this only gets relevant with the next few commits that restore labels and phandles. Note further that this rework is necessary with these further changes, because phandle markers are currently not considered for type guessing and so a phandle at an offset that isn't a multiple of 4 triggers an assertion if the property was guessed to have type TYPE_UINT32. Now that guess_value_type() is only called for data chunks without markers, the function can be simplified a bit. Signed-off-by: Uwe Kleine-König <u.kleine-koenig@baylibre.com>

If the input has a __symbols__ node, restore the named labels for the respective nodes. Signed-off-by: Uwe Kleine-König <u.kleine-koenig@baylibre.com>

The __local_fixups__ node contains information about phandles. Parse it to improve the result when decompiling a device tree blob. Note that this is essential to do before the __local_fixups__ node is removed in generate_local_fixups_tree() because otherwise the information contained in that node is lost. Fixes: 915daad ("Start with empty __local_fixups__ and __fixups__ nodes") Signed-off-by: Uwe Kleine-König <u.kleine-koenig@baylibre.com>

The __fixups__ node contains information about labels. Parse its properties to create phandle markers which improve the resulting dts when decompiling a device tree blob. Note that this is essential to do before the __fixups__ node is removed in generate_fixups_tree() because otherwise the information contained in that node is lost. Fixes: 915daad ("Start with empty __local_fixups__ and __fixups__ nodes") Signed-off-by: Uwe Kleine-König <u.kleine-koenig@baylibre.com>

The need for the plugin flag is determined by the existence of __fixups__ or __local_fixups__. This is a bit simplifying because if __fixups__ or __local_fixups__ exist but don't have properties, the plugin flag isn't needed. But in practise the test should be good enough such that this corner case doesn't matter. Signed-off-by: Uwe Kleine-König <u.kleine-koenig@baylibre.com>

ukleinek · 2025-07-04T08:00:12Z

There is one test failure when the top commit is missing because restoring phandles from __fixups__ of an overlay results in a dangling phandle which isn't compilable without /plugin/'. But it's only the top commit that adds /plugin/. Reordering doesn't help, because then again we suffer from the test failure that we already saw before when you tried to apply the "Set DTSF_PLUGIN if needed when compiling from dtb" without restoring data from __fixups__ and __local_fixups__.

dgibson · 2025-07-15T02:27:35Z

This commit uncovers a problem that we already have since commit 915daad ("Start with empty local_fixups and fixups nodes"):
$ cat test.dts 
/dts-v1/;
/plugin/;

/ {
	node {
		property = <0xffffffff>, "string";
	};
	
	__fixups__ {
		somenode = "/node:property:0";
	};
};

$ dtc -I dts -O dts test.dts 
/dts-v1/;

/ {

	node {
		property = <0xffffffff>, "string";
	};
};
I'll think about a fix.

Ouch, ok. I missed that nasty side effect of 915daad. Unfortunately I don't think the fix proposed in the latest version is quite sufficient.

It's an important principle in dtc's design that it should generally be safe and lossless to convert dtb -> dts -> dtb. Likewise it should be "lossless" to convert dts -> dts, in the sense that going directly dts -> dtb and going dts -> dts -> dtb should result in the same output. This is exactly what those test cases are designed for.

This is supposed to be true even if the input tree contains garbage. It needs the basic structure to be parseable, of course, but if the properties contain things that are semantically malformed, they should be passed through unchanged. 915daad broke that.

Your latest series partially fixes this, but not completely: it will still simply delete entries it doesn't understand in the input tree. It might also re-order entries that it does understand. I can see two ways to fully fix this:

Change fixup generation, so that instead of unconditionally adding the new fixup, it first checks to see if a matching fixup is already in __fixups__ or __local_fixups__. With that in place, 915daad can be reverted. That fix could be applied immediately, then your code to understand the existing fixups could be done on top of that. Alas, that is kind of messy and a fair bit of work.
This approach might be a bit easier... but I'm not sure it gets every case correct. Instead of wiping __fixups__ clean as a single step, you remove entries one at a time as you parse them into fixups in the live tree (so you now know they'll be re-generated). This could still result in re-orderings which isn't great, but maybe acceptable. I'm also not sure if it handles all breakages from 915daad that don't involve a dtb -> dts stage.

ukleinek · 2025-09-19T10:11:28Z

FTR: I took this patch set to the mailing list as I prefer this workflow after this PR got unwieldy.
v1: https://lore.kernel.org/devicetree-compiler/cover.1755692822.git.u.kleine-koenig@baylibre.com
v2: https://lore.kernel.org/devicetree-compiler/20250919092912.663304-8-u.kleine-koenig@baylibre.com

ukleinek mentioned this pull request Oct 21, 2024

Device tree overlays cleanup analogdevicesinc/linux#2624

Merged

ukleinek mentioned this pull request Oct 21, 2024

Some phandle related improvements and some bug fixes #93

Closed

dgibson requested changes Oct 28, 2024

View reviewed changes

dgibson mentioned this pull request Oct 30, 2024

Improve dtdiff by avoiding or nulling the phandles #145

Open

ukleinek force-pushed the restore_phandles branch from 845fb23 to 07f279e Compare January 12, 2025 01:20

dgibson requested changes Jan 13, 2025

View reviewed changes

treesource.c Outdated Show resolved Hide resolved

treesource.c Outdated Show resolved Hide resolved

dtc.c Outdated Show resolved Hide resolved

ukleinek force-pushed the restore_phandles branch from 07f279e to 5c04a2d Compare January 14, 2025 11:04

ukleinek force-pushed the restore_phandles branch from 5c04a2d to 9827002 Compare January 14, 2025 11:35

dgibson reviewed Jan 24, 2025

View reviewed changes

treesource.c Show resolved Hide resolved

tests/over-determined.dts Show resolved Hide resolved

treesource.c Outdated Show resolved Hide resolved

treesource.c Outdated Show resolved Hide resolved

ukleinek force-pushed the restore_phandles branch from 9827002 to c21b18f Compare January 25, 2025 15:46

ukleinek mentioned this pull request Jul 1, 2025

dtdiff: Use input format dtb for dtbo files #168

Merged

Emit /plugin/ when compiling to .dts with DTSF_PLUGIN set

a9b9a47

This fixes `dtc -I dts -O dts` to make the file a plugin if the source file is one. Signed-off-by: Uwe Kleine-König <u.kleine-koenig@baylibre.com>

ukleinek force-pushed the restore_phandles branch from c21b18f to 3442d0b Compare July 1, 2025 11:00

ukleinek requested a review from dgibson July 1, 2025 15:40

dgibson reviewed Jul 2, 2025

View reviewed changes

treesource.c Show resolved Hide resolved

dgibson requested changes Jul 2, 2025

View reviewed changes

dgibson reviewed Jul 2, 2025

View reviewed changes

tests/run_tests.sh Outdated Show resolved Hide resolved

dgibson reviewed Jul 2, 2025

View reviewed changes

tests/run_tests.sh Outdated Show resolved Hide resolved

ukleinek force-pushed the restore_phandles branch from 3442d0b to 4f025c1 Compare July 2, 2025 09:44

ukleinek added 4 commits July 2, 2025 11:56

New helper to add markers

7001053

The add_marker() function is used to create a new marker and add it at the right spot to the relevant marker list. Use it in the add_string_markers() helper (which gets slightly quicker by it). Signed-off-by: Uwe Kleine-König <u.kleine-koenig@baylibre.com>

Restore labels from __symbols__ node

6a6f5ae

If the input has a __symbols__ node, restore the named labels for the respective nodes. Signed-off-by: Uwe Kleine-König <u.kleine-koenig@baylibre.com>

ukleinek force-pushed the restore_phandles branch from 4f025c1 to fab7c49 Compare July 2, 2025 10:06

ukleinek added 3 commits July 2, 2025 13:05

ukleinek force-pushed the restore_phandles branch from fab7c49 to 64bbc33 Compare July 4, 2025 07:36

ukleinek mentioned this pull request Aug 1, 2025

Handling of __fixups__ and __local_fixups__ removes precious data #170

Closed

dgibson force-pushed the main branch from 41369ff to cba90ce Compare August 18, 2025 23:33

dgibson force-pushed the main branch from 849947a to 8d15a63 Compare January 28, 2026 01:14

Conversation

ukleinek commented Oct 21, 2024

Uh oh!

ukleinek commented Oct 21, 2024

Uh oh!

ukleinek commented Oct 24, 2024

Uh oh!

ukleinek commented Oct 24, 2024

Uh oh!

dgibson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dgibson commented Oct 28, 2024

Uh oh!

ukleinek commented Oct 28, 2024

Uh oh!

dgibson commented Oct 28, 2024

Uh oh!

ukleinek commented Oct 28, 2024

Uh oh!

dgibson commented Oct 30, 2024

Uh oh!

ukleinek commented Jan 12, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ukleinek commented Jan 14, 2025

Uh oh!

ukleinek commented Jan 15, 2025

Uh oh!

dgibson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ukleinek commented Jan 25, 2025

Uh oh!

ukleinek commented Jul 1, 2025

Uh oh!

ukleinek commented Jul 1, 2025

Uh oh!

dgibson commented Jul 2, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ukleinek commented Jul 2, 2025 •

edited

Loading