decoder2: rework decoding of sumtypes

[Larsimusrex]

Reworks the sumtype decoder to allow for more complex sumtypes and better error handling.

New Features

• Adds Decoding for json2.Null, so that json2.Any is fully supported
• Sumtype that cant be resolved now gives a clearer error e.g. could not resolve sumtype Sum, got string_ instead of Expected number, but got string_
• map & array now only get resolved if the entire type matches (instead of just using the first array / map). This allows for sumtypes with multiple different array types like type MultiArray = []int | []bool. This also enhances error messages.
• Options & Type aliases are still not support, but now give runtime errors instead of cryptic compiler errors
• Fix random compiler error when having multiple sumtypes with maps in the code

BREAKS

Because the decoder now gives errors, instead of resolving bogus sumtypes, sumtypes with structs behave differently from json and json2. e.g.

pub type Animal = Cat | Dog

pub struct Cat {
	cat_name string
}

pub struct Dog {
	dog_name string
}

// before
decoder2.decode[Animal]('{"cat_name": "Tom"}') // Animal(Cat{'Tom'})
decoder2.decode[Animal]('{"dog_name": "Tom"}') // Animal(Cat{})
decoder2.decode[Animal]('{"dog_name": "Rex", "_type": "Dog"}') // Animal(Dog{'Rex'})


// after
decoder2.decode[Animal]('{"cat_name": "Tom"}') // will now throw error, missing field _type
decoder2.decode[Animal]('{"dog_name": "Tom"}') // will now throw error, missing field _type
decoder2.decode[Animal]('{"dog_name": "Rex", "_type": "Dog"}') // Animal(Dog{'Rex'})

[huly-for-github]

Connected to Huly®: V_0.6-23428

[enghitalo]

I don't think that json.Null should be supported by decoder2. You can use optional instead. And I've been wanting to move Any to another library like datatype library instead of json for a while now

[enghitalo]

To avoid the divergence break between the json, json2 and decoder2 libraries, I think it is worth investing in integrating decoder2 with json2. It won't be long before this becomes possible and it would avoid a huge headache. You are doing a very dedicated job and I am sure you will succeed

[Larsimusrex]

I remove the json2.Null decoding and allowed options to decode null as none, so now you can do:

type NewAny = int | string | bool | []NewAny | map[string]NewAny | ?int

decoder2.decode[NewAny]('{"name": null, "value": [0, 1, "hi"]}')!
// NewAny({'name': NewAny(Option(none)), 'value': NewAny([NewAny(0), NewAny(1), NewAny('hi')])})

I also made some minor fixes to the checker, so that it no longer throws errors for {"arr": []} and for strings containing carriage returns.

[Larsimusrex]

To avoid the divergence break between the json, json2 and decoder2 libraries, I think it is worth investing in integrating decoder2 with json2. It won't be long before this becomes possible and it would avoid a huge headache. You are doing a very dedicated job and I am sure you will succeed

As for integrating into json2, I only found 2 things that don't work. The first one is that the number conversion will not correctly checks types (will try to convert -1.92 into a u8 with no error), which is easy to fix.
The second one is the decoding of arrays of options, e.g. []?int which I don't think is possible with the current implementation of generics (fn [T](arr []T) will always lose the option, while fn [T](arr []?T) will always add one). On the other hand the json2 decoder just uses the default value instead of none so this behavior would break anyways.

[Larsimusrex]

I also removed all references to json2.Any so import json2 is not needed anymore.

[enghitalo]

@felipensp @spytheman I think there's a feature or discussion about every sumtype being optional, right? How can I test whether a sumtype has been initialized or not?

[enghitalo]

If not, I think we should think about it a bit. I think it's a bit dangerous for decoders to be forced to choose a type when the decoded value has no defined type

[enghitalo]

For example, what happened here, nothing tells me that the null came from an integer value. But I also don't see how this could have been done better.

assert json.decode[Maybes]('null')! == Maybes(?int(none))

[spytheman]

I think there's a feature or discussion about every sumtype being optional, right? How can I test whether a sumtype has been initialized or not?

There is not. Sumtype values are not options. The default value for a sumtype, is the default value for the first variant in the sumtype list.

[spytheman]

In any case, JSON on its own lacks the concept of sumtypes, so we are free to implement whatever encoding/decoding for them that we want to.

[Larsimusrex]

I still think the changes in this commit are correct, options should decode null as none and decoder2 should not have any references to null or any types.

I plan to implement custom decoders anyways, so you could do

struct Null implements CustomNullDecoder {}
type Any = string | f64 | bool | []Any | map[string]Any | Null

without any references to Null or Any types.

[Larsimusrex]

custom decoders could be used for time.Time or math.big too.

[enghitalo]

Personally speaking, I'm against using Any within decoder2 as if it were different from any other sumtype. I think that what works for Any, should work for any other sumtype. To keep the json library simple and avoid unexpected behavior

[enghitalo]

Good work!!!

In case you're interested, I always ran benchmarks vlib/v/tests/bench/bench_json_vs_json2.v or vlib/x/json2/decoder2/tests/bench.v and post the results in the PR to track performance

[spytheman]

@Larsimusrex I also think this is excellent work. Thank you.

[spytheman]

@enghitalo