In this episode, Yehonathan Sharvit shares with us what "Data Oriented Programming" is and how it can improve the way we code and help us reduce the number of bugs in our code.
He also talks about his new book "Data Oriented Programming" and his peculiar way of explaining such a complex topic.
Get in touch with Yehonathan:
Get the book!
Check out "Data Oriented Programming" and use this code to get a 35% discount during checkout: pod20minjs22
Don't forget to leave a review of the episode or the entire podcast on Podchasers!
Meet our host, OpenReplay:
OpenReplay is an open-source session replay suite, built for developers and self-hosted for full control over your customer data. If you're looking for a way to understand how your users interact with your application, check out OpenReplay.
In the same way then that when you add the two, a number, let's say 42, you have two, then you get 44, but you don't mutate 42 42 stays 42 in Data Oriented programming, the same thing happened with objects or hash maps. If you have a hash maps with two keys, two fields, and you want to add. You don't mutate the original map you create a new one. And if you adere to that, if you separate codes from dataFernando Doglio:
Okay. Uh, first of all, thank you, Fernando, for inviting me to your show. It's a pleasure to be with you. Um, my name is Yehonathan. I live in Israel in a small village. But as you may hear, uh, in my accent, I was born in France. I have a high, strong accent, and, uh, I, I, I, I have lived in Israel since I am a 17. So I learned at school here and I work. I have worked only in Israel as a software engineer. Uh, Maybe the most interesting, interesting thing I can say about my career is that there is definitely a turning point at 10 years ago. And this turning point happened when I discovered closure before closure, uh, programming was just my job. And since closure it has become my passion. Um, so for the, you, are you familiar with.Fernando Doglio:
I am a bit, I had played around with it, nothing professional though.Yehonathan Sharvit:
All right. Yeah. I was going to ask you if closure had anything to do with data oriented programming. Interesting. Okay. I definitely agree with you. Looking into different ways of coding, different languages, especially if there's follow such a different, uh, paradigm than the one you normally use. Uh, it's a eye-opener because even if you're not going to be using it on your day to day, it will give you different ways of solving the same problems. So, absolutely agree with that. All right. So after that intro, uh, tell us finally, what is data oriented programming?Yehonathan Sharvit:
Interesting. Okay. So data on one site code on the other, that's the philosophy.Yehonathan Sharvit:
Yes. And that, data has a value data as a value, right? Yes, immutable.Fernando Doglio:
Okay. And is this the same as data oriented design? Because I've been, I've been trying to research a bit about the topic and the both, both terms came up. So yeah.Yehonathan Sharvit:
Yeah, no, no, it's not. It's unfortunate, but you know, naming is hard and, uh, it's not just, data oriented design is about, is about optimizing performance and data oriented programming is about. Reducing complexity system complexity. It makes your code easier to read and to maintain and to debug, et cetera. It doesn't make your code run faster.Fernando Doglio:
Okay. And from what I've been reading, uh, in your book and a bit online, uh, from what I understand is, you shape your data based on your, uh, business requirements. Right. And then write the code to, to deal with it. Would you say that that is a better approach that letting the business requirement shape your, your logic essentially, and then that logic uh, shape whatever data you need essentially bottom up versus a top down approach?Yehonathan Sharvit:
Hmm. Uh, okay. So, so far, let me recap, uh, summarize here. You talked about immutable data. You talked about general purpose functions, uh, and data manipulation manipulations. So how are we not talking about functional programming essentially? What's, what's the difference between functional programming and data oriented programming? They seem very similar.Yehonathan Sharvit:
Yeah, that's a great question. So, first of all, What they have in common is that we separate code from data. Secondly, data is immutable, but usually in functional programming, usually when we say functional programming, we mean Haskell, oCaml F#, all this stuff. And we, with all these family functions, we spend lots of time typing our data. And data is not generic. We don't represent business entities as hash maps, represent business entities as custom types. And that's not the case in data oriented programming. So if we take functional programming and you remove types, you get data oriented programming.Fernando Doglio:
And it makes, it makes it, it makes, it seems like a small thing, not to type, but it makes a huge, huge difference. And also in functional programming. We don't emphasize it so much. The importance of immutable generic data structure we clone, or we do custom tricks to create a new version of data in data oriented programming, we leverage a smart data structure called persistent data structures that are, uh, efficient. So when you create a new version of the hash map, let's say with a 10 fields, you don't clone the hash map. There is a way, a smart way to, in a sense, cut the cake and have it too you create a new version without creating a new version. So we have two different versions separated, but the data that is common is shared and because it's immutable, it's safe to share. And that's usually not the case in when we say functional programming in the traditional way, the traditional sense.Fernando Doglio:
Cool. And on that topic on the book you mentioned that one of the key functionalities, one of the key traits of object oriented programming is inheritance. And, uh, you see you, you propose the problem of how do we deal with inheritance when. We don't have objects. We don't have classes. Um, you talk about the multi method. Can you describe that? Because if it's a very, uh, interesting solution that I've never really heard about until after your book. So, uh, can you tell our audience what is a multi method and how do you use it, especially related to the DOP?Yehonathan Sharvit:
Absolutely. I think that is the curse of functional programming, right? I mean, up, for some reason object oriented programming is way more popular. So we all tend to think in objects, at least that's the way we are. We're thought to start at the start of our careers, anyway, many, many of us eventually discover functional programming. And are like you.Yehonathan Sharvit:
Th there is a great quote by Alan Perlis is about that. So it's, simplicity, simplicity, never preceeds complexity. Simplicity always follows complexity. So first we find the solutions that are complex, we use them and then we're able to distill and to simplify, and then we find a simpler solution. But the problem is that there's complex solutions that already widespread. Always too late in a sense.Fernando Doglio:
It's true. That's true. But, uh, yeah, I love that quote. I'm I'm actually going to steal it because it's very good. It makes a lot of sense. Okay. One thing that I've not been able to really understand about DOP is. If we've been talking so far, uh, and, and I understood until now we split data one side and logic and code on the other. You have a very interesting diagram, your book about that. Showing that that's split when you create a data and generics and validation and all that is split from the actual logic. But if you do end up having an, you know, a real world application would have, uh, complex data structures, right? Uh, your function. that are meant to be generic will have to know anyway, the internal structure of that data so that they can use it because there is no really functionality associated to the data. So doesn't that couple, the actual data to the implementation, to the point where they're not really different from, uh, or they can't really exist individually. Uh isn't that kind of what Object Oriented Programming ends up being?Yehonathan Sharvit:
Yeah, so that's a great question. And let me first tackle the easy part. So first of all, you have functions that don't care about. They done the structure of the data. For example, the function that let's say. Counts the number of fields in the hash map, you could pass any hash map and the function works, you could think about even more advanced function, like a function that rename keys. You could have a function that receive a map and a field named transformation. And in order to implement this function, you don't need any knowledge, but they don't have structure of the data. And you have lots of functions like that. And in object oriented programming, you cannot write functions like that because you always have to know the exact type of the fields. You cannot write, uh, an object method that renames fields it's impossible. It makes no sense. Or you would have to use reflection or very complicated thing. While when you separate data from code, it's trivial. And many of the lodash set of functions are advanced functions like that, that are pretty hard to implement and impulsive in a DOP and impossible to implement in OOP. So that's, that was the easy answer now to the difficult answer, let's take an example. Very simple example. Let's say you want to write a function that calculates the full name of an author. Okay. Let's say we are in a library system and we are, we have books and authors and users and members, et cetera. And let's say we represent an author as a map with the first name field and the full and the last name field. And we want to write a function that returns the concatenation of the first name and the last name, nicely capitalized with the space. So now you cannot write this function as a generic function. You need to know what's the name of the field? Is it first name? Is it the first name is the F lowercase uppercase. So you need this information, um, on the one hand. Yes. But on the other hand, you could, once you have written the function, you could pass to the function, not only authors, but you could pass also users. If the user is represented as a map. With first name and last name and many other fields, email, and the number of books, et cetera, the function that calculate the full name doesn't care. So that's the power of data oriented programming. It only cares that about that. Some fields need to be there. It does not care about the whole, it's not limited to a type of maps. It works with all maps, where the keys, the fields, first name, and last name are there and there are strings and that makes a huge difference. First of all, in terms of reusage capability. And secondly, in terms of testing, if you want to test the function, uh, full name, you don't need to know about authors. You don't need to import the class definition of an author. You just need to create a map with a first name and the last name and in the real life, when the application runs the map that gets passed to the function full name may have dozens of other fields and you don't care. You don't have while in OOP if you had a function like that, you would have to create a real author object or a real user object with all the fields, just to test a function that cares about two fields. There is, uh, an idiom that goes, and it's a bit demagogic, but it says that in OOP usually in order to test a tiny function. You need to build your whole system wide in data oriented programming. If you, if you need to test the function that checks the first time and the last time you only need to pass the first time and the last name as a map.Fernando Doglio:
And by the way we have the complimentary problem is that sometimes our code is so dynamic and so flexible that we don't know what we have in hand. So for example, if I, uh, go back to the full name function that calculate the full name of an author or a user, there is no clear way, but by just looking at the function signature to know what is the expected shape of the data, you have to read the code to discover that first name needs to be written like that. And there are three ways to deal with that. First of all, to say, okay, that's a problem that the first way, the second way is to document the function and somewhere to write, these function expects a piece of data, having a field called first name in the field called last name. And the third is to leverage, to type "a la carte". So you could use a language like TypeScript and type this function, full name and say. The data pass here should be, uh, should be compliant with an interface that has a first name and the last name and both of the fields need to be string, or you could leverage a technology like JSON schema, where you would specify the schema of the data somewhere and just say, write a piece of code that validates that at run time. Whereas in TypeScript is at compile time that validated the data is compliant to the expected schema.Fernando Doglio:
Okay. Yeah, absolutely. If you don't want to go to the TypeScript way then JSON schema, and I think you use that in your book as well. That example is the way to go.Yehonathan Sharvit:
About yeah. And the important thing, no matter if you JSON schema or TypeScript. The important thing is that it's "a la carte". And for example, if you write a function, renamed keys or count fields there, you don't need to types because there it works with any data. So it's, again, it gives you much much more flexibility than the traditional, a statically typed approach, where you have to type every single piece of code, or sometime you just want to prototype. And you just want to experiment with code. You don't want to waste time on typing. So. It's the responsibility of the developer to decide what to type and what not to type. And sometimes it's a problem because as another idiom goes with great power comes great responsibility.Fernando Doglio:
I know that one. Okay. Awesome. Final question about DOP.. Would you recommend it for any use case or have you found a use case where, or type of projects or type of platforms of software, any way that you've tried to build with DOP that you said, well, this is definitely not the best way to go about it?Yehonathan Sharvit:
I don't think it's a solution for everything. I think it's a good fit for information systems, for systems that manipulate data that comes from the outside. For example, a backend application, the data does not come from the application. Usually it comes from that from a database or from a message queues or from a rest API or from it comes from the outside or a front end application that receives data from the backend. But if you build a compiler or if build a game engine, And you generate, you create the data about. The AST abstract syntax tree of the code or about the mechanics of the game engine. And you have no surprise, you create the data that you consumed, then it makes it makes more sense to use another, uh, paradigm, uh, where you don't need as much flexibility as we need when we build the information systems.Fernando Doglio:
Okay. Finally, can you, uh, tell the audience a bit more about the book and what's in it and what they will get out of it?Yehonathan Sharvit:
So, first of all, let me tell you something, uh, that you have probably noticed. And I'm curious to know if you appreciated it or not. The book is written as a story.Fernando Doglio:
Yeah. I saw that. Yeah. Yeah, it's at first it was interesting. I had to like adjust to, to the way it was written because I couldn't just pinpoint the specific topic I wanted to, uh, I wanted to, to, you know, to read about, but at the same time, since this was a completely new topic for me. Reading the whole dialogue between these two characters, uh, helped me ease into the topic and understand, you know, what you, uh, the reasoning behind it and, and kind of what you want it to, to explain. And the, the counter examples that you gave. So it was, uh, I never seen a book, a technical written like that. So it was, it was interesting.Yehonathan Sharvit:
Yeah. Yeah. Uh, it's really fun to read and it's, it's easy to understand the topic, so definitely recommend it. And we're going to have a discount code for our readers on the show notes so they can pick it up with an interest in discount. Thanks to Manning for that. Um, okay. So. Enough of DOP. I'll have three more questions that we ask all our guests. So I want to ask you, what is the best advice you've ever received in your career?Yehonathan Sharvit:
Yeah, the best advice I received is when I want to explain something to someone always start from a concrete example, always, always, always a concrete example that speaks to the person in front of me. And from this example slowly by slowly. Uh, step up into the abstraction and always make sure and confirm that the person stayed with me and understand not before a, before being aware of this advice, I use to start from theory and abstract stuff and claims, and it never worked for me. And when I started to, to speak concretely with concrete example, It's much easier for me to connect to people and to go where they are instead of bringing them into my realm. I go down into their, uh, their stuff.Fernando Doglio:
Interesting. Okay. And it looks like you use that on the book as well. Very, very good advice. Uh, what's the most exciting project you worked on?Yehonathan Sharvit:
Yeah, no, absolutely. I mean, that's the smartest way to go about it if you asked me because there is already a lot of work already done, there is no need to reinvent the wheel every time. Um, final question. What is one thing you wish you knew when you started coding.Yehonathan Sharvit:
All right. That's it. Thank you so much. Uh, for being here, please tell our listeners where they can find you, and if you're working on something else, something you want plug it's your, it's your time.Yehonathan Sharvit:
So I'm quite active on Twitter. My Twitter handle is @viebel. V I E B E L . I have a blog it's blog.klipse.Tech B L O G dot K L I P S E dot T E C H. If you, if you purchase the book from the Manning website. There is a forum live forum. So you can ask me questions there, or you can ask on Twitter or you can send me an email. It's easy to find my email online. Um, right now I'm taking a break. I've completed the book and it's going to be printed very, very soon. I really enjoy the process of writing a book right now. I don't have any idea for a second book, but, uh, One idea maybe would be to address the, what you mentioned, Fernando, that it's not easy to see the principles and the highlights of the book. So maybe I will write a non story, a version of the book and who knows. I'm waiting for the inspiration that will hopefully come for a new idea.Fernando Doglio:
Yeah. Cool. Absolutely. Uh, if it doesn't then just rest because writing a book is not easy. I should know. Uh, so, um, enjoy, enjoy the time the break the, and that is, that is all we have. That is all the time we have. Thank you again so much for coming in for speaking about this to me and the audience. And that's it. See you guys on the next one. And thank you again for being here.