feature/issue-175-auto

[njlr]

• Introduce Auto module
• Re-enable many tests

This PR introduces an Auto module that is agnostic to the JSON backend used. It does this by composing Encoders / Decoders from Thoth.Json.Core using reflection. There is some Fable specific stuff to pave over the limitation of reflection for some targets.

The majority of the old tests are re-enabled and unchanged.

LMK what you think!

Edit: See #232

[njlr]

Fixed merge conflicts

[MangelMaxime]

Thank you for the update, I just got back from vacation will have a look to make a new Thoth.Json release this week and take this chance to look at the Auto API support once again.

[MangelMaxime]

Hello @njlr,

I am finally starting to have a look at this PR seriously.

Looking at the code it seems like there are a lot of changes compared to the originals versions, do you remember what motivated these changes?

For example, you seems to use this pattern a lot:

#if FABLE_COMPILER
                let fail (innerType: Type) (x: string) : obj =
                    Decode.fail x |> box
#else
                let private failGenericMethodDefinition =
                    getGenericMethodDefinition "Fail"

                let fail (innerType: Type) (x: string) : obj =
                    failGenericMethodDefinition
                        .MakeGenericMethod([| innerType |])
                        .Invoke(null, [| x |])
#endif

instead of probably the DecoderCrate

    type private DecoderCrate<'T>(dec: Decoder<'T>) =
        inherit BoxedDecoder()
        override __.Decode(path, token) =
            match dec path token with
            | Ok v -> Ok(box v)
            | Error er -> Error er
        member __.UnboxedDecoder = dec

    let boxDecoder (d: Decoder<'T>): BoxedDecoder =
        DecoderCrate(d) :> BoxedDecoder

    let unboxDecoder<'T> (d: BoxedDecoder): Decoder<'T> =
        (d :?> DecoderCrate<'T>).UnboxedDecoder

Is it because it improves something over the crate?

[njlr]

Looking at the code it seems like there are a lot of changes compared to the originals versions, do you remember what motivated these changes?

Sorry, it has been a while!

IIRC, the thinking is to move all reflection and dynamic casts to the encoder/decoder generation step and out of the encode/decode stage. In theory this offers better performance for the most common use-case: generate an encoder/decoder once and then call it many times.

The Auto module uses reflection to invoke the core API in an automatic way, but does not contain any special encoder/decoder types.

[MangelMaxime]

Ok thank you for the answer.

At least now, Thoth.Json has benchmark setup so it should be possible to make a quick comparaison between both approach an see which one works the best at least for .NET.

[reinux]

If you ever get a chance, this would be pretty nice to have. Using some simple workarounds to get extra coders working on both Fable and .NET 9 right now.

[njlr]

If you ever get a chance, this would be pretty nice to have. Using some simple workarounds to get extra coders working on both Fable and .NET 9 right now.

Sorry, I had forgotten about this PR but agree it would still be useful!

@MangelMaxime What are the next steps needed?

[MangelMaxime]

Originally, I wanted to make a benchmark comparison between your version of the Auto decoders and the old ways of doing them.

But I don't think it matters too much to me, I am not focused on performance for Thoth.Json.Core. I mean the new API is slower than the previous one because of its modularity but that's also its strength.

---

I always wanted to get this PR merged, but always kept postponing it ... I think we all have this kind of PR siting around that we "fear" and I need to face it 🤣

Let's make a plan for releasing Auto API support with Thoth.Json.Core API.

1. I need to review this PR, and make sure it works. I think the easiest way for me would be to copy the changes manually to a new branch, so I can enable the tests one by one. Here I have trouble reviewing them because they have the formatting applied...

   However, it means that @njlr you will no longer be the author of the changes... Are you fine with that?

2. I want to check if the new API can support fix the following issues because they can require some underlying rework:

   • Allow user to customize even the default encoders #57
   • vNext - Check behaviour of nested option when adding lossles option especially with skipNullField = true  #89
   • vNext - Check if user can override behaviour of generic type #88
   • Extra encoder doesn't get called for type with generic params #169
   • Check what happen when mixing manual and auto style when using DUs #87
   • vNext - DUs with a single Case but multiple arguments aren't encoded correctly #86

[reinux]

Thanks for doing this!

I agree, I'm also happy to do some caching if that's what I need to do to enjoy Thoth's ergonomics.

[njlr]

Originally, I wanted to make a benchmark comparison between your version of the Auto decoders and the old ways of doing them.

But I don't think it matters too much to me, I am not focused on performance for Thoth.Json.Core. I mean the new API is slower than the previous one because of its modularity but that's also its strength.

With .NET 10 optimizations, it may actaully be faster. This would need to be measured, however.
I also did some experiements with "out" variables in Core API to avoid allocations that improve performance greatly, but that's another story...

I always wanted to get this PR merged, but always kept postponing it ... I think we all have this kind of PR siting around that we "fear" and I need to face it 🤣

I know the feeling!

Let's make a plan for releasing Auto API support with Thoth.Json.Core API.

I was hoping that the Auto API could live in a different package so that compilation targets with limited reflection support can use the Core API.

1. I need to review this PR, and make sure it works. I think the easiest way for me would be to copy the changes manually to a new branch, so I can enable the tests one by one. Here I have trouble reviewing them because they have the formatting applied...

   However, it means that @njlr you will no longer be the author of the changes... Are you fine with that?

I don't mind who the author is, whatever is easiest to make progress 🙂

2. I want to check if the new API can support fix the following issues because they can require some underlying rework:

   • Allow user to customize even the default encoders #57
   • vNext - Check behaviour of nested option when adding lossles option especially with skipNullField = true #89
   • vNext - Check if user can override behaviour of generic type #88
   • Extra encoder doesn't get called for type with generic params #169
   • Check what happen when mixing manual and auto style when using DUs #87
   • vNext - DUs with a single Case but multiple arguments aren't encoded correctly #86

At a glance, I think all of these are fixable under the new approach.

[MangelMaxime]

Let's make a plan for releasing Auto API support with Thoth.Json.Core API.

I was hoping that the Auto API could live in a different package so that compilation targets with limited reflection support can use the Core API.

I probably wrongly explained my self, the Auto API will be in it own packages yes.

When I talk about Thoth.Json.Core API this is to make a distinction between the legacy and the new APIs.

[njlr]

I probably wrongly explained my self, the Auto API will be in it own packages yes.

I wonder - could we merge this as-is and release Auto API under an experimental pre-1.0 version so people can use at their own risk?

Meanwhile we can start working through the tickets you listed above and shake out any issues?

[njlr]

I am unsure why the CI is failing on Windows.

This works locally:

# nix develop

./build/sh test

Edit:
Success!
Needed to update the lock files

[MangelMaxime]

This PR is superseded by #232

#232 is a manual port from this PR with some performance optimization and more code sharing for better maintainability

[goswinr]

This PR is superseded by #189

@MangelMaxime did you mean superseded by #232 ? I am looking forward to use it.

[MangelMaxime]

@goswinr Yes sorry, this is probably why the link were not working.

It is now out at as Thoth.Json.Core.Auto 0.1.0