JSON is everywhere. As developers, we will be required to work with it at some point or another. Maybe in the embedded space you could avoid it, but as a cloud and web developer, I have no choice but to be intricately familiar with all things JSON. As an enjoyer of statically typed programming languages, I need some way of deserializing the JSON payload into a custom defined type in my language of choice. The techniques talked about here are opinions I hold myself that I have developed over my career and work well for me. I wanted to share these ideas with you so that you can also experiment with different ways to see what works best for you. Since I started my career as a C# developer, that’s where we’ll start here but then move onto what I believe are now greener pastures in JSON deserialization when looking at functional programming languages such as Gleam.
Using C# reflection to deserialze JSON into a C# object
In 99% of cases, you’ll either be using Newtonsoft.Json or System.Text.Json to work with JSON. The standard way to work with these libraries is to define what’s commonly called a POCO, or plain old C# object, that mimics the structure of the outgoing or incoming JSON payload. It has always been good coding conventions to use separate types that are used exclusively for this purpose instead of reusing e.g. domain objects. It falls right in line with separation of concerns, as well as a clean abstraction for the serialization and deserialization layer from the other layers.
As a quick example, to deserialize the following JSON:
{
"id": 10,
"name": "Lynn Conway",
"documentIds": [42, 745, 12, 55, 234],
"isManager": true,
"address": {
"street": "Main st.",
"number": "3",
"city": "Maplon"
}
}we would generally write two POCOs. One for the “Person”, and another for the “Address” subobject.
public class Person
{
public int Id { get; set; }
public string Name { get; set; }
public IList<int> DocumentIds { get; set; }
public bool IsManager { get; set; }
public Address Address { get; set; }
}
public class Address
{
public string Street { get; set; }
public string Number { get; set; }
public string City { get; set; }
}With the most simple way to deserializing that JSON looking something like this:
using System.Text.Json;
public async Task<Person> GetPerson(int id)
{
var http = new HttpClient();
var json = await http.GetStringAsync($"https://exampleapi.dev/person/{id}");
// The magic line that uses Reflection
var dto = JsonSerializer.Deserialize<PersonDto>(json);
var person = MapToDomainPerson(dto);
return person;
}Both Newtonsoft.Json and System.Text.Json work by inspecting the C# property names using reflection. Meaning that at runtime they query the object type definitions to get the a string representation of the property names and use this to match up against the fields in the JSON payload. This is why it’s important to mimic both the structure as well as have exactly the same property names in the C# objects as they are in the JSON.
You’ve probably written dozens of similar code blocks in all the various languages, because this is the primary way of communicating between services. This also isn’t unique to C#, as most statically typed languages will follow a very similar pattern. Deserializing JSON in this manner works exactly as you’d expect, and both the people behind Newtonsoft as well as Microsoft have put in years of work to ensure that these deserialization methods are as performant as possible. Reflection based deserialization is a well established pattern that has stood the test of time.
This is also a great way to work with JSON for most use cases, as for many projects you or your team will be in control of both sides of the API. This means that you can agree to certain conventions that work best for the programming languages you use, as well as how you expect more complicated objects to be modeled and handled. However, there will also be times when it won’t be this simple. Often times you’ll encounter JSON payloads from services outside of your control, or at least unable to change the format for backwards compatibility reasons. Developers coming from different ecosystems with different coding conventions will have different ideas on what the JSON format should look like. Even with our small example above, one of these convention differences can be seen. In JSON, it’s common to name the properties using “camelCase” convention, but in C# it’s common to use “PascalCase”. This specific difference is so common that it’s actually the default behaviour of the reflection based mapping to automatically convert between the two.
Working with strange JSON payloads
Occasionally, you’ll encounter JSON that looks completely off. Oftentimes this is because you’re working with third party APIs that you don’t control. In the worst cases, working with these APIs can become a serious pain as they seem to throw all conventions out the window. Take a look at the following example.
{
"Data Points": [
[{ "2022-12-19 08:14:00": 0 }, { "2022-12-19 10:00:00": 114 }],
[{ "2022-12-19 09:00:00": 48 }],
[{ "2022-12-20 15:00:00": 81 }, { "2022-12-20 15:44:00": 0 }, { "2022-12-20 13:00:00": 384 }],
[{ "2022-12-20 15:44:00": 0 }, { "2022-12-20 14:00:00": 238 }]
],
"Postal codes": [
43212,
63453,
"07556",
"01221"
]
}We immediately see some problems. The most obvious one is that the object keys don’t follow any programming naming conventions. Worst of all, they contain gasp spaces. Examining this further, the “Data Points” object is a list of subobjects, which could be seen as a list of tuples, or a list of individual key value pairs. And to make matters worse, the “Postal codes” object is a list of mixed types of numbers and strings.
The problematic object keys have the simplest solution. You can apply an attribute to the C# property telling the deserializer how to map it.
public class MeasurementDto
{
[JsonPropertyName("Data Points")]
public IList<IList<(DateTime, int)>> DataPoints { get; set; }
[JsonPropertyName("Postal codes")]
public IList<string> PostalCodes { get; set; }
}Using the JsonPropertyName attribute is also a relatively common pattern, however the JSON payload still wont’ deserialize into this POCO properly. If you look closely at those DateTime keys, they actually don’t conform to any of the accepted formats by the C# DateTime struct. Additionally, neither of the aforementioned deserializers actually know how to convert one of those single key JSON objects into a single type such as a C# tuple. This part will fail silently and you end up with the default value of the defined tuple.
The PostalCodes property will also fail as C# can’t map the numbers into strings. It won’t implicitly convert them behind the scenes. What you’d actually need to do is either write a custom type converter, or use a strict superset type, which in this case is simply object with which you’d throw all type safety out the window.
They way to handle these cases is to write a custom type converter, and then configure the properties to use those instead. e.g. [JsonConverter(typeof(DataPointTupleConverter))] for the DataPoints. The implementation of the DataPointTupleConvert is very involved and requires adhering to a specific interface.
All of this is very doable, but you slowly lose the comfort and ease-of-use with a plain and simple reflection based POCO. Additionally, usage of these custom attributes and custom type converters is hidden in the POCO definition, which hides the complexity of these types at the call site of the deserialization. Some might argue that this abstraction gives the deserialization and definition of the POCO a clean separation, but in my opinion this makes debugging strange edge cases more difficult.
Take care when using refactoring tools
This might seem like a minor issue, but it has bitten me a select few times. When using some refactoring tools such as JetBrains Rider’s Rename Refactoring, it will “helpfully” suggest to search in string literals as well for the property name that you want to rename.
The problem with is is that it will also find the custom defined JsonPropertyName configured in the attribute and rename it there, silently breaking your deserialization configuration. I have made this mistake a few times in the past, and therefore always turn off that feature.
Admittedly, Rider also shows you what it will change before actually making the change, but in certain scenarios, especially when many files get updated, it’s easy to miss the one spot where you actually don’t want it renamed. You could of course argue that your integration tests should catch this, and I would agree. But we all know that integration tests aren’t perfect, and nobody has perfect branch/property name coverage.
Looking at Gleam for a better way
In Gleam, the primary way of working with JSON is by using coders. A set of function to either encode a type to a JSON representation, or decode, to parse a JSON string into a defined Gleam type. The basic idea is to write functions that manually decode the JSON payload, and compose with a set of decoders to assign each JSON property a specific primitive type, and then using these parsed values you would build up your custom type.
I wouldn’t be writing a blog post about parsing untyped data into a statically typed language without mentioning Parse, don’t validate.
Functional programming has generally preferred this way of dealing with JSON, as the primary primitive to the JSON parsing is, like in all things functional programming, just a function. A function that can be composed and passed around as values.
As an example, we want to deserialze the “Person” JSON from above into the following Gleam record:
pub type Address {
Address(street: String, number: String, city: String)
}
pub type Person {
Person(
id: Int,
name: String,
document_ids: List(Int),
is_manager: Bool,
address: Address,
)
}
import gleam/dynamic/decode
fn address_decoder() -> decode.Decoder(Address) {
use street <- decode.field("street", decode.string)
use number <- decode.field("number", decode.string)
use city <- decode.field("city", decode.string)
decode.success(Address(street:, number:, city:))
}
pub fn person_decoder() -> decode.Decoder(Person) {
use id <- decode.field("id", decode.int)
use name <- decode.field("name", decode.string)
use document_ids <- decode.field("document_ids", decode.list(decode.int))
use is_manager <- decode.field("is_manager", decode.bool)
use address <- decode.field("address", address_decoder())
decode.success(Person(id:, name:, document_ids:, is_manager:, address:))
}
import gleam/json // must include gleam_json package
fn read_response() -> Result(Person, json.DecodeError) {
let response = "json response from above"
json.parse(from: response, using: person_decoder())
}At first glance, this seems like an incredibly error prone and involved way to parse JSON. And you’d be right (or not, see below). There’s a ton of boilerplate code required to map every single field in an expected JSON payload and decode it using a specific decoder for each and every field. The main takeaway is that Gleam doesn’t have any metaprogramming capabilities (yet) to do something like C# with reflection so you’re stuck composing these decoder function endlessly. However, I would argue that writing the C# “PersonDto” POCO above requires a roughly equal amount of boilerplate code. You’re writing a property and defining its type for every field you wish to map out of the JSON. This holds true for both cases. But the Gleam approach comes with additional benefits.
Combining decoding and domain mapping
In most applications, you have specific domain types that drive your entire application. It’s also good practice to separate your domain types and your JSON deserialization types. If we take a look at the strange JSON payload example above, it has “Data Points” and “Postal codes” separated under different properties. But for our domain we want each postal code to have its own set of data points. A simple C# example would look like this:
public class Measurement
{
public string PostalCode { get; set; }
public IList<(DateTime, int)> Measurements { get; set; }
}Now the “MeasurementDto” class from above that we’re using to deserialize the JSON into requires some explicit mapping logic to be able to construct a “Measurement” object out of it. Using one of those automatic mapping libraries won’t do the trick here anymore. So for this (contrived) example, we were required to create several custom JSON deserialization converters, and then also write some custom mapping logic to be able to construct the domain type out of it. All of this extra work for a “simple” reflection based deserialization approach.
Looking towards Gleam again, or any approach that uses decoders, we can combine the decoding and mapping into a single operation. Since we don’t need an intermediate type and are simply decoding the primitives directly, we can therefore then immediately construct the appropriate domain type. Here’s an example of how this could be achieved in Gleam:
import gleam/time/timestamp // must include gleam_time package
import gleam/json // must include gleam_json package
import gleam/dynamic/decode
pub type Measurement {
Measurement(
postal_code: String,
measurements: List(#(timestamp.Timestamp, Int)),
)
}
// There's a bug in this implementation. See if you can spot it and fix it
fn timestamp_count_decoder() -> decode.Decoder(#(timestamp.Timestamp, Int)) {
decode.dict(decode.string, decode.int)
|> decode.map(fn(date_count) {
date_count
|> dict.to_list()
|> list.map(fn(part) {
timestamp.parse_rfc3339(
part.0 |> string.replace(" ", "T") |> string.append("Z"),
)
|> result.map(fn(t) { #(t, part.1) })
})
|> list.first()
|> result.flatten()
|> result.unwrap(#(timestamp.system_time(), 0))
})
}
fn measurements_decoder() -> decode.Decoder(List(Measurement)) {
use data_points <- decode.field(
"Data Points",
decode.list(decode.list(timestamp_count_decoder())),
)
use postal_codes <- decode.field(
"Postal codes",
decode.list(
decode.one_of(decode.string, or: [decode.int |> decode.map(int.to_string)]),
),
)
let measurements =
list.map2(data_points, postal_codes, fn(dp, pc) {
Measurement(postal_code: pc, measurements: dp)
})
decode.success(measurements)
}
echo json.parse(from: "strange json payload from above", using: measurements_decoder())There are several things happening here. We first have the same domain type as in C# defined at the top. We also have two decode functions, but the “timestamp_count_decoder” is only used by the “measurements_decoder” although it could be used on its own as well (something about everything just being composable functions :D). The strange property names are of no concern as the first parameter to the “decode.field” functions is a string that must match the property name exactly. Looking at the “postal_codes” decoder first, here we already see a simple example of the power of composable functions. Using “decode.one_of” we can supply two different decoding functions, and Gleam will figure out which one to use depending on what it encounters in the JSON payload.
Secondly, there’s the “data_points” decoder. This function does several things. The data in the JSON needs to be parsed as a dict first, and then converted into tuples. But there’s also the part about having to parse the timestamp into Gleam timestamps. This is all done inside the “decode.map” callback function. Similarly, the timestamp parsing returns a “Result” type that needs to be unwrapped as well. But the entire pipeline still returns a single decoder that we can use to decode the list of data points.
Finally, we can construct the domain type by mapping over both lists at the same time and matching the postal codes to the data points by index (yes, we have to hope that this weird third party API sticks to that convention). Now that we have a list of “Measurement”s, we can finally tell the entire decoder that we have successfully decoded a list of them with the “decode.success” function. So we’ve combined the deserialization, as well as the mapping into a domain type in a single decoder that also gives us type safe error handling.
A reminder about railway oriented programming
As previously mentioned, we’re parsing the incoming payload into a usable type. But the entire mechanism in the Gleam version makes use of the “Result(TOk, TError)” type. I’ve previously written about railway oriented programming (although using F#), and to me, decoding JSON like this just manages to fit so neatly into everything else that makes functional programming great. Everything is just a function.
Gleam even has LSP support
As previously mentioned, writing those decoders looks like it’s a huge hassle with a tremendous amount of boilerplate that needs to be written. Luckily, the Gleam LSP has a code action that generate coders for you. The code action is good enough to give you a quick start for the coders, but won’t generate a full complex object graph as that includes some implementation details for you to figure out. But it offload some of the hassle of writing these things. Yet another reason why Gleam is such a nice language 🙂
