Insect report:

The following:


Code:
" ô q pq p X  Delta ->Str8


throws the following error on building:
Quote:
" ô q pq p X Delta ->Str8


But the following works fine:

Code:
"   q pq p X  Delta ->Str8
I have the same issue with every accented character.
xlibman wrote:
Insect report:

The following:


Code:
" ô q pq p X  Delta ->Str8


throws the following error on building:
Quote:
" ô q pq p X Delta ->Str8


But the following works fine:

Code:
"   q pq p X  Delta ->Str8
I have the same issue with every accented character.
I'm able to replicate this error, including when the characters are selected using the token pane. I'm not sure the precise status of PT_'s backend overhaul, but I suspect there's some encoding problem happening.
Hey there, I noticed the token parser/compiler seems to have some round trip issues.
If you export this program, it will print the two letters "pi"

Code:
Disp "\p\i

But if you then import the resulting .8xp, it loads as this

Code:
Disp "pi

And if you export that one, it prints the symbol "π" instead. In fact, this happens for all tokens whose string form is made up of other tokens'. Is this a known issue? Is there a way I can solve my programs getting changed when loading for this reason? Thanks!!
The way the tokenizer is set up, this sort of thing is difficult to fix (it works basically the same way as Tokens, down to using the same XML token data). Since the token data is a mapping from tokenized bytes to strings, the detokenization is generated by reversing that mapping and there's no mechanism to customize it.

More generally, I feel like the better solution to the "pi" sorts of problem would be to escape the version that you want to translate, so you might say "\pi" to output the π token and remove ambiguity from the textual version of the code.

Do you know precisely which other tokens are problematic as well? I imagine things like colors could be problematic since the string "Red" might be turned into a token that's not supported by monochrome calculators, but the solution there seems like ensuring capability levels are correctly flagged. Am I overlooking something?
I can definitely escape the text version no problem, but when importing a premade 8xp, it doesn't get escaped, so information is lost when loading in.
As far as I can tell, all tokens that can be made with other tokens are affected. Like if you spelled out "ClrHome" manually with letters in an .8xp file, SC3 would think it was the command once it was loaded.
tryashtar wrote:
As far as I can tell, all tokens that can be made with other tokens are affected. Like if you spelled out "ClrHome" manually with letters in an .8xp file, SC3 would think it was the command once it was loaded.
Is that a problem? I can imagine some use cases where you wouldn't be using a program as actual TI-BASIC where you might care about that, but if it's actually a BASIC program then doing that transformation seems strictly beneficial, since the token is smaller and should be displayed the same.
Tari wrote:
Is that a problem? I can imagine some use cases where you wouldn't be using a program as actual TI-BASIC where you might care about that, but if it's actually a BASIC program then doing that transformation seems strictly beneficial, since the token is smaller and should be displayed the same.
its not as easy as it seems, as I've had to deal with that with tivars_lib_cpp too:
- It may not be displayed the same depending on the language Smile
- Which is fine for non-strings since it will actually be a command, but not for strings... so a quick solution is not to maximal-munch the token when the tokenizer state is currently within a string
- ... but there are also special cases for that with the CE Send( function that IIRC needs tokenized stuff inside for string interpolation to work. Well, there's something that's going on anyway, that TI-connect CE does (even though it doesn't even solve point 1 for some reason), but I forgot what special treatment it was.

This last point is actually particularly hard to solve because even even if Send("Foobar is simple to detect, Send(Str1 would mean that the tokenizer would need to make sure Str1 contains tokenized symbols contrary to the solution of the list item 2 above. Anyway, for now this specific issue is just not handled by any lib/software afaik
For the ClrHome example it would be beneficial to swap it out. It's not beneficial when the token's string representation in SC3 doesn't match its string representation on the calculator, like "pi" turning into "π" and "sqrt(" turning into "√(". I think pi is the most troublesome example because of all the normal english words that happen to have the letters pi in them
Maybe there could be an option in SourceCoder during importing that disables the feature that turns strings of letters into tokens when those tokens are located inside quotes? That could be handy when importing programs. I often find myself having to manually add plenty of \ characters all accross my code everytime I do because of how it messes up with the sub() string command.

However, an easy workaround for this is to avoid using lowercase letters.
I made a parser that corrects this by inserting a character whenever the loaded symbol would cause the stream of text to be interpreted a different way going forward. (I use tab instead of backslash since backslash is a token in this case)

https://gist.github.com/tryashtar/592ad3848281f0cd4d2d812ee7efafb4

It's garbage but successfully recreates my collection of 8xp files identically byte-for-byte round trip.
I noticed another one that's even worse: I use "->" a lot to display an arrow. With this issue, that converts into →, and trying to print that is a syntax error
tryashtar wrote:
I noticed another one that's even worse: I use "->" a lot to display an arrow. With this issue, that converts into →, and trying to print that is a syntax error

This isn’t a bug. You need to use a backslash in there so sourcecoder knows you’re not trying to use the store token. So, it needs to be “-\>”
Michael2_3B wrote:
tryashtar wrote:
I noticed another one that's even worse: I use "->" a lot to display an arrow. With this issue, that converts into →, and trying to print that is a syntax error

This isn’t a bug. You need to use a backslash in there so sourcecoder knows you’re not trying to use the store token. So, it needs to be “-\>”


The issue isn't just you sometimes need to add a \ when writing source code, it's that when importing an 8xp file, it doesn't include a \ in the correct places. This doesn't cause issues if you do all of your development on a computer, since it's easy enough to add a \ once for each relevant token you add. But if you alternate back and forth between editing on a computer and on-calc, you need to manually add all of the \s back each time you move from the calculator to a computer.
Using tryashtar's hack above as inspiration, I've implemented escaping in SourceCoder so it will insert escapes to disambiguate strings that could be ambiguous.

The test cases I checked with were "pi is not p\ie" and "-\>", each of which round-trip correctly through an 8xp file.
Legend! Thank you so much!! <3
Apparently, SourceCoder refuses to detokenize prgms I've downloaded off of the Cemetech Archives. I attempted to load Calcuzap (curious what the code looked like) and it said "Disassembly failed at line (whatever it was -- I'll have to check)". Any thoughts?

I did this on a fairly new (3-4 yrs. old) Mac desktop, on Chrome (or possibly Safari), both o which are on their latest version. The computer is running the latest version of macOS Apple had distributed at that point (this was a few weeks, maybe a month ago at the outside).
Calcuzap is an assembly game and thus cannot be detokenized - The source is included in the zip file however?
Oh hi again peoples \o

So I'm playing around with coding up a thing on my old 84+, doing the coding and testing in SourceCoder, and I've found out that you can send an appvar to the emulator, but you can't retrieve it, gives you "Invalid Variable Name" error.

I forgot to include, the appvar has lowercase letters in the name, which is what is causing the problem. The name of the appvar I'm using is "Reclass" which causes everything after the c to be a token.
Indeed, SourceCoder does not handle lowercase variable names properly. I recommend using all-uppercase variable names until/unless we eventually fix this.
Sharing doesn't work. (I can access my file's code from share links but everyone says they can't)
  
Register to Join the Conversation
Have your own thoughts to add to this or any other topic? Want to ask a question, offer a suggestion, share your own programs and projects, upload a file to the file archives, get help with calculator and computer programming, or simply chat with like-minded coders and tech and calculator enthusiasts via the site-wide AJAX SAX widget? Registration for a free Cemetech account only takes a minute.

» Go to Registration page
» Goto page Previous  1, 2, 3 ... 29, 30, 31, 32  Next
» View previous topic :: View next topic  
Page 30 of 32
» All times are UTC - 5 Hours
 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

 

Advertisement