Introducing BeeSon ~ bee + json

hello I'm Logan and we prepared with the explanation video for a completely developmentally data topic called the psalm and it's just released its first version the zero one zero it has properties such as the edges and valuesome random access the object's Keys have in exact order for serialization and last but not least it supports proper binary marker inclusion proofs on container elements first have a look on what is the most usual civilization of Json objects here are two Json objects with variable names Json and json2 with the very same values but the order of the elements are different if I want to serialize this objects I have to encode these objects into Strings characters that exactly corresponds to Json notations on the example you can see even though the two objects share the same values their civilizations are different uh not equal to each other and so to speak bite-wise I cannot tell the civilizations who are the same values moreover even if it kept the key order but I add some values to the body survey for example the following values are shifted by the length of the inserted elements because of this it loses its Random Access property in other words I cannot tell in which place I can read out the value of the each property in this example only if I start to interpret the Json objects sequentially from the start this Random Access property is required to efficiently provide inclusion proofs for values in the binary marker tree the real origin story behind this samples that in Monterey one zero proposal the data structure design already had Random Access for all of all of its elements but not in on its metadata that was serialized in the same way as mentioned before a Json in order to understand a bit more uh the spinal marker tree for short PMT let's look let's have a look at the fundamentals section when the data has Random Access then the lookup is the fastest for its elements if this kind of data is represented in the BMT one can prove the element in question is included in the data blob identified with the content addressed 32 byte segment this is called BMT inclusion proof that requires data only logarithmical to the length of the volt data blob so this is how the BMT looks like the bottomless level is the data serialization itself that the data slice up into 32 bytes of of segments each level has half amount of elements then delivered below uh do they have to calculate the white elements here of this BMT is hashing the corresponding two segments together from the below level for example in order to get each zero in the first level the d0 and D1 have to be have to be hashed together the hashing organs will do this until it results in a BMT root hash One Singular 32 bytes this segment encodes this sub sound original data by BMT hashing algorithm which gives the content addressed property for this identifier the BMT like this has a 128 segments in the data level in spawn which is four kilobytes and that is called a chunk in order to get this address only this pen has to has to be hashed together with the with the BMT root hash where the span holds the byte size of the data substant under the address I have to State here there is no four kilobytes of limitation for for data blobs because of this since you can use multiple chunks in order to represent a full blob the hedging function being used in BMT is the ketchok 256 its output is 32 bytes that defines the smallest unit in a BMT that I referred with the name segment the slightest change in the input data will result in in a completely different hash output the hashing function is also extensively used I mean this function extensively used in the ethereum ecosystem as well with the proper fundamentals we can dive into the theory behind of a bison let's discuss first what types of the bson handles by definition a Json value can be either a string number object or a pulan or null here the object and array acts as a act as a container types since they just include other elements in their defined structure nevertheless in byte representation of types you have to specify more more of these types for example what kind of number you store such as in 32 or float 64. now you can see in this list what kind of types pson can handle currently with their corresponding notation such as null has value 1 Boolean has two and so forth the numbering doesn't follow an incrementing scheme because later type values can be mixed creating another types for example an an in32 array that will have value 8 305. array an object have a new level version then the elements can take null values and also there are mistypes like song CAC and the song COC that are 32 bytes of sworn hash references one exotic type is the supervision the tanker solicit other container types but about that later and now we reached a point where I can elaborate the structure and the logic behind the marshalling and have the sterilization happens it consists of three parts header describes the version and the type of the piece on value the type specification details the properties of the type it is required only at container types and the data implementation the value of the psalm basically these bits and components are fitting into voice segments meaning even if the information can be expressed lesser than 32 bytes it will be potted anyway into a vote segment starting at the beginning with the header the header is always searched device of which the first part is the version bytes but the first byte is a result for the Beason identifier that follows by three bytes of this semantic version of the used Visa the second part is the block flags of which the last two bytes are representing the type as it was also mentioned before it has 26 bytes of padding in order to fit into a boa segment which in this case can be used for extending the base types domain set later it is known what the header is and since the type specification is only required that container types you can see an example for a known container type with a data implementation the examples like this are using the JavaScript library of bsan the library exports a piece on class that can be instantiated with any type of Json value for Digital Library generates the corresponding piece and attributes and configuration as an example for a Json string where we initialize the piece and object with with a Json string in a variable named B song Bond it has multiple methods but we just need the serialization that returns 64 bytes it starts with uh version bytes one zero one zero which represents it is a piece on blob and it has a 0.0 version after the 64 bytes of padding we see what piston type we have which is zero eight from the type table it is shown the the eight indicates a string value the last 32 bytes are the segment fitted string that who has the characters for decentralized and a quote Mark that indicates the end of the string I'll put the padding for the last time the Beeson is suited for the decentralized storage ethereum swarm its smallest units is four kilobyte for which you have to pay even if your data is lesser than four kilobytes if is significant and Standalone business component is segment fitted then the BMT inclusion proof is simple and straightforward for more details about PMT check our another video link in the description in case of container or mistypes the types of education is also needed since even if the base type is an array at its structure still must be defined take a glance on the first example for Aries I have such a structure looks like it starts with the type definition size in two bytes the states have long the type definition array is that is colored in Gray one type definition is six bytes always so I can calculate how long the Wall textures equation is in bytes so six times the type definition size uh in this example the super definition reference size is zero and then later I will detail it let's see how it looks like in action that I have a Json array with three string types at the first three this is and the number type for the last division type has 8192 value for type that indicates it is an area type and it also has four type definitions describing its elements by this we see that this array has a structure that the states how long the array is in other words data implementations omis must give values for for these four elements in the described types after the header the yellow rectangle shows the type definition size which is four then the super based on count which is zero for now and there is the first type definition in Gray which has two parts the type of the first element and the segment length that reserves the segments in the data implementation to set value for this element in this case the type is eight so it is a string and this segment length is 1 meaning the value of this first element will be stored in the third segment index of the data implementation also under introduces an upper limit for string length so a string longer than 32 bytes cannot be represented here if more bytes are needed for setting sting value in data implementations this segment length has to be increased emphasized previously the array type specification States the length of the array in which elements cannot be noise the null is a particular value since it represents some elements doesn't have a defined value in standardization this new value for attributes have to be expressed somehow the solution of bson is having a new label bit Vector that handles as many bits as the size of data definition array which is indicated with m each bit in this bit Vector represents a type definition element in the same location and indicates whether it can be null or not the new label containers I require another bit Vector at the beginning of the data implementation called new speed vector the noise bit Vector has the same size as the nullable bit vector and those bits indicate which elements of the objects are news the type value of the new label array is 8448 and the new label bit vectors value is in bluish with value 7. in binary representation it is one one that means all attributes can take null values the data implementation starts with the before mentioned the new speed Vector that indicates which indices of the container are null values if the if a bit equals to 1 mean by the corresponding new label bit Vector speed equals to zero DBS and data implementation is invalid objects are not really different from arrays in bison those just have an additional schema notation for keys as you see it has additional field called markers length that was the bite length of the marker sorry where all object keys are stored the marker is placed after the type definition array and it has all objects keys concatenated in all in the order of the type definitions in yellow the marker length is a serialized that has a value of 17 since the object Keys each body's ID name have length 16 plus 1 for the trailing code mark follows by the type definitions array of which elements are different than in bson arrays because it has to contain how many bytes must be popped from the start of the mark cursory to get this Associated key the first element has a three biased length of marker so from the marker array the first three bytes will be popped which holds the characters each on this level we only know the body's property is an array but its exact structure is unknown the type definition of the array is a detailed in the corresponding data implementation part of the array value in other words one type specification just specify the properties on the same level but not below where it is defined that is a way have to encode any arbitrary complex structure of a container type into a single segment and this is called supervisor the idea leverages the content addressing in a way that it only makes the references for the Visa header and the type specification by that the data structure elevates the resolving of the test specification to the decentralization algorithm if the P sun has a superbisson type on the topmost level that sophistication will be replaced by a singular supervisor reference this reference can be resolved by the compiler by a simple mapping from the reference to the header and the task specification bytes after this it can process the data implementation in the same way like at the other container types as mentioned it can encode any structure below the described level as well by using super type references in the type specification then Super type references is are used for describing a container type element its types education also doesn't have to be serialized in the data implementation section super type references also what header plus type specification bytes and it's a type specification also can have super type references and so on and so on because of this the topmost superb Vision reference includes all sub substance superb type preferences thereby every types you can see on the right there is the supervision object on the left is the simple container type the red area will be replaced by the green in case of supervisor and all bytes below is the data implementation part by that only seven segments are needed to prove the exact data structure of a blob for four kilobytes of data that equals to a single segment proof of a data blob so it is structure proof can be cheaper since this supervision is a specific type it must be indicated somehow for the compiler that the container type is a superbism for that the JavaScript library has Type managers for each element of the piece of object where a superbism flag can be set to true this can be done also by modifying the DNA object of the Bison type nature which is the Giza representation of the piece and configuration in this example the top level object is a supervision as it's it is indicated at the bottom but the body's attribute at this second index can also have this flag set through in this space type specification object if this flag is true then the compiler will replace the volume type specification data into its swarm hash and on load it will resolve this form hash into a die specification with all of these Concepts we achieved to express the structure of any Json object in 64 bytes which is followed by its values in the data implementation section so a file represented by a swarm hash can consists of multiple chunks but for the sake of Simplicity let's sum up what happens in a single chunk scenario for inclusion proofs on BMT inclusion proof one must provide the segment to be proved its segment index and the corresponding system segments in order to result in the desired PMT root hash by hashing those together by the rules of dbmt hashing the structure is expressed by two segments in case of supervision and those segments are pairs so by inclusion proofing the first segment the another segment also will be proved for four kilobytes the PMT has seven levels so we must provide seven sister segments included with the supervisor reference to prove this structure by this the leftmost side of the bmt3 is known of course most of the cases it is not enough to prove only this because some fraud proof also has to be provided uh in the data blob for which the data segment also must be passed that it doesn't obey to some rules in its value these rules can be written in Block validator smart contracts where the validators can digitalize every attribute of this supervision schema or positive way to another validator smart contract let's say we have a structure that describes a person and drawers defined by small contract such as the each property has to be above 18.

by the supervisor reference the proof can be handed over to its corresponding validator that has already the offsets and types of attributes in the data implementation this validator contract will this write this segment if the BMT root hash of the inclusion proof data equals to the challenged hash after this only a compensation is needed against the value like this value has to be above 18 and in in the previously mentioned example uh if the data segment L is in the lower level of the structure the root level validator will be will pass the validation to the corresponding validator contract based on the super bson reference otherwise if the lower level element is not a supervision the validator has to interpret the elements type specification this is how an inclusion proof system can work on any complex adjacent structure if you want to try it out you can download it from npm this JavaScript library with which we create these examples as well or build this Library itself from building the source code that you can reach out in the under the FED data Society peace on GitHub repository thank you and see you next time