Tracking mapping keys with the Truffle Debugger by Harry Altman

everyone I'm going to be talking about truffle debugger and how keeps track of mapping keys so this is the thing with truffle debugger - right so the solidity language famously does not know what a mapping keys are and in the debugger when we are displaying variables at a given point in the transaction we want to be able to show what is contained in the mappings this requires knowing what these are and well that's not possible in general but we can at least keep track of the keys that were used in a given transaction and we can even do this when we've got you know ested mappings like here I've got this function that does a bunch of crazy things with mappings nested inside other things but if we run this into the bugger and we get to the end it's all there you've got this map inside this struct inside this array and you've got you've got all this but that's actually the the nested map in think that's actually not what I want to focus on right now close that down I want to focus on something that looks a lot simpler let's look at this test this sets up a bunch of things in mappings that look very straightforward we've just got you know we've got bulling mappings and bytes mapping some integer mappings and address mapping and you know we skip to the end just a moment there we can see it's it's all there um but the surprising thing is that um let me close this down the surprising thing is that keeping track of all this is a bit more complicated than you might expect well so I mean the first question is how do you do it at all and the basic answer um which I've sort of typed up on the right here on the Left I've got the relevant part of code which I'm afraid is rather opaque but uh you'll eel you'll see there's enough I'm not this enough so the basic version is that you know so the trouble debugger in order to tell where we are in the code right it uses the source maps that sublimity provides right we have the bytecode we have the source map we know where we are in the bytecode we nuts we can find where we are in the source and then we can the other thing it uses is the ast the abstract syntax tree representation of the source that solidity provides so we have where we are in the source we can then map that to a particular node in the abstract syntax tree and we can there therefore use various information about that note so the basic version of how this works is that whenever we are on a node and are processing the note through and it corresponds to an expression we store the top word stack as that node finishes and we associate it to that node as well as to the current stack frame and similar stuff and I said the top chord of the stack some types you know use a more than the word on the stack really it's the top and word stack is appropriate one or two and then whenever we get to an index axis for a mapping we're accessing one of those mapping values right we decode we look at the expression that is the index and we decode it and then we say okay that's our math and key let's associate it to that mapping so this is pretty simple and it works fairly well um it's uh like I said it's a pretty good system that Nick came up with but there are some wrinkles in it um and so first off what do I mean when I say associated to that mapping here's come on yeah yeah I have sort of a text slide here what when I say Associates that mapping there an early version associated Samantha's ast node that obviously has various problems so okay so I mean associated to the address and the storage slot how do we get the storage slot same mechanism when we pass over the node that is the correspond to the mapping itself that we're indexing into rebus toward the top word of the step back and you know that's not that's our pointer and we actually do a bit more than that in order to keep track of some other information but I don't want to go into that here because then I would not there then I would have time for a song okay so you have this basic version um and now we start getting to the wrinkles and the reason that handling all these things it's not actually so simple so oh boy I'm kind of about to uh spill the secret sauce here I guess but hey our debugger is a open sauce so uh no secret anyway so what about these string literals here that doesn't seem like a problem they're just string literals right um except thing is think about the the process I described for how we handle these things in order to get the information for the index node we have to step through that node at some point there has to be some actual EDM instruction in the bytecode that is source math to that range in the code and for some notes that doesn't happen and string literals are an example of this I say string literals I mean also hex literals right I mean those are considered by celebrity BIC to be essentially the same thing and so what's our solution to this so let me actually go into the code here into the code math oh yeah we've got this tiny bit of code here for cleaning boolean to handle out of range boolean's make sure they can't read it it's true I'm not gonna go any more into that um and oh by the way you may notice why is there a loop here we would be coding math and keys right it's just like what I said doesn't involve any sort of looping process why is there a week we're gonna get to that but one thing at a time so in order to handle string literals we've got the special thing we're before the main case down here this this here is the main case but before that we've got this special case the candle string literals and actually some other things if you go ahead and read the comments I've written there where if it's our if our index is a simple constant which includes string literals we don't use the stack at all we've just read it we could read the information straight out of the ast this object here is one of our internal pointer objects that we use for indicating data locations such as a place a spot on the stack a spot in memory and whatever and in this case it represents not a spot in memory or storage or ever but a spot in the ast that we are going to read data out of okay that lets us handle string literals um so there here's our revised process we're gonna do an exit level every time you process a node we're gonna put the top word of the stack or top two words and then we can aim x-axis we're gonna read the data off the stack or the lip for the index and we're going to be coming back and that's gonna be our mapping key except for string literals but then we'll read it EF key itself but oh they're next promise when I read that not all types to take up just one word on the fact that's easy enough to handle sometimes sake of two words on set I already mentioned that let's give it okay so we've got a reprise person what I get by there's another problem um decoding pointers so I I skipped over this before I said oh let's just decode the data from the stack now what if it's a point there to memory or to storage or to call data well by itself that's not a problem our decoder know how to handle that it's it it knows to go look up the appropriate information in memory or store for call data but there is a little bit of a problem with making the shirt it knows the right type here so let's if we look at the cone here we see this um crazy thing where I've got a splice definition um sorry some of these names are a little out of date misleading definition here might be a better thought that you know we've got a spliced know where we take two different ast know then we kind of splice them together and why on earth would we do that well we wanted to code according right so we were your code right obviously celebrity's a statically typed language so if you wanted to code do we need to know in advance what type we're decoding and if we have the mapping and it's keys are strings it will be reported as its keys being in India's he will say excuse our memory strings but we might be giving it to a pointer to a story string or recall beta string now in general we wanted to code according to the type of the mappings keys right not the type of the particular expression because there might be you know implicit conversion right so North Dakota for two type of the Matthews keys but if the type of map that Key says memory straight we're actually dealing with a storage string or a call data string we're going to get nonsense the decoder is going to repoint areas entirely the wrong type of pointer it's going to look an entire in the wrong place so we have this sort of crazy splicing thing where we take the he the node forbid that defines the Mac and keys and splice onto it the location of the actual index expression that's where we've got this splice location function here all right all right we've got that we've got that we've handled that you've got a revised process here you're gonna in decode the word from the stack but the string literals we're going to read the value from the ast instead and also we have to do this splice defect but what if the key is a constant state variable because remember what I said earlier is that in order for our process to work the he has to be um source mapped to at some point there must be some actual right you have to actually step through that key and for constant state variables this doesn't happen now if the constant state variable is part of a larger expression that's the key this is fine but if that is the entire key it's just going to get skipped over and it won't work so what we have to do is here is the case for handling constant state variables this one is kind of a messin nightmare to some extent it's a constant state variable what we have to do is we have to look up the definition of that compensator but we have to look at the node that's where we have you have to look at that index node and we have to say oh this is a constant state variable and it's definition is over here and look at that ast node and then we can read the information out of that assuming it's a fairly simple Thompson anyway if you do something crazy in your constant definition we might run into some trouble and we might fail to keep track of the cake but I'm assuming that's a fairly minor case regardless I like to be comprehensive so we can handle you know most constant state variables okay okay you've got we can handle ordinary cheese and the pointers handle string literals we can't cancel canto state variables all right what if the key is a hexadecimal literal and the key type is a 2-bike then like what if it's being used as a bytes and rather than as an integer which you know is something that I you know I made sure best here and now like why would this why would this be a problem at all well here's the thing I have two decimal literal from someone whose point of view that's an integer so when we put the key when the raw key beta on the stack he's gonna go on the stack as an integer left Pat it right right aligned so if you do this with 0xff here it's going to go on the staff to 0xff on the end here but this is a byte not an integer and those are our right padded wrap the line the actual value of the mapping key w as you fix so if we were if we didn't have any special handling we would get we would put on the wrong map and he and we would I mean we'd be looking in the wrong place I mean you know the the decoder would just be like it would just look at this first byte here would be like oh I guess the mapping key is 0 X 0 0 and that's not correct oh we go back to the code here you may notice you may have noticed a while back when I was talking about string literals the code here talks about it's not just for handling string literals it's it for handling hexadecimal literals - and there's some special stuff in the decoder so that it knows that when it's reading a I mean I say hexadecimal but most really occurs any numeric literal um that's but it knows when it's reading a numeric literal from the ASD so the string a numeric literals will read them out of the a s teens that are from the sack and it knows when it's doing this that if it's decoding as a bike's and it has to shift it appropriately okay okay can ask a question uh right can ask a question yes can you go back to the back to the inside so you said that the constant is put right aligned onto the stack is it then discarded when the mapping location is computed no of course not obviously it has to be left shifted in order to be used but okay we have s right alone okay done yeah yes it's not yes you right that obviously it has to be left shift in order to be used so obviously goes on to the stack of that at some point but the question is does it go on to that at the stack at the point you can capture this actually gets into some stuff I skipped over so normally we assume we get to know that the last map struction for that note that the next thing on the stack is what we want there's actually some cases where that's not true because it goes into unmapped code and so we have to look ahead to the next map instruct math instruction when I say map that unmapped I really mean like you know map 2-1 you know we solidity internal stuff and so it's actually possible that this could be have been handled by that system I haven't checked that that system actually came later chronologically so I never thought to check that regardless in this case reading it out of the AST is how we could do handle it it's possible we could handle it instead by looking a little bit ahead instead but I haven't checked right but I mean the problem has to be the problem is that it has to be at a point that we can detect and know aha this is where it is so yeah anyway there is one remaining problem and you'll notice I still have not explained the the answer to the question why on earth is there a loop here so let's get down to the bottom of this loop and you will see why there is a loop here or a better way of saying this is the example so here I have this address map and for a while what what happened is this first assignment this first assignment would work fine we keep track of time but the second one wouldn't my thought is there a problem addresses no probably nothing to do with addresses problem has to do with certain type conversions after this doesn't work as a mapping key or didn't and he does now obviously but it didn't work as a mapping key because the address conversion from contract at the EVM level bit to Noah there's no actual EDM instruction that corresponds to the address conversion so there's no instruction to get source map to it which means once again no value gets stored so and you can see this with other types of versions as well as well so like white one that would work fine because that type conversion involves a shape this is sort of the butt address this or eat one those are no ops so the problem would occur and in sufficiently old solidity there was unary Plus which is another no op um so fugly this is kind of the opposite problem with the hexadecimal literal problem right so this this problem is where there is a conversion in the source code but no actual instruction in the EDM whereas the hexadecimal little problem was there was a conversion instruction in the EDM but no corresponding node in the source code unfortunately that's the only type conversion type type of type conversion where that happens but there's several sorts of type conversions where this happens where it's at the EVM level up to Noah so how do we handle this well if our decoding has failed thus far and the node type for the index expression is a type conversion we have to make sure that it has failed this far because if you know if it's a type conversion that isn't the Noah then we don't want to do this but in that case you know things will it for it already so but if it's the type if it's if it's if things have failed thus far and it's a type conversion then we look inside and we look inside the arguments that type conversion and we try again and similarly with unary plus if you're using really old Lydia's people look inside the unary plus and then we will go back to the top of the loop and try the whole thing again and that's how we were able to handle these no op type conversions and so with this revised that's why there's a loop in map and key decoding and with this revised process with all these wrinkles we are paved people to handle and keep track goes basically any mapping fee you can throw at us except maybe really complicated constant state variables but that's not a comment we can handle based pretty much any map and key you can throw out and that is how the truffle debugger doesn't