Updating GameTZ's software
25-Jan-23 11:45am
#1 bill
I'm planning to update/swap-in an alternative piece of software that GameTZ uses to serve web pages.
Hopefully, it will just work and no one will even notice.
But, there may be issues. So, let me know if you notice things not working or seeming off.
I'm not sure when I'll do this, but perhaps off-and-on over the next few days, perhaps starting soon.
For people interested in the technical details:
For 20+ years GameTZ has used FCGI the Perl module that implements FastCGI. But, after the trouble last week with quotes getting turned into ?s because of character set issues, I realized FCGI does not support multibyte characters, and thus is essentially obsolete.
I found something newer (only 10 years old!) called PSGI that supports UTF-8 character sets (e.g multibyte characters). PSGI can basically be swapped in for FastCGI (using an emulation module).
I have been testing a PSGI-based server component and it seems to work well (on my test server). The component is called Starman (which is unfortunate because I've had that David Bowie song in my head all week because of it).
Once I put Starman on the live site, it will get a lot more usage and be truly tested. I may have to fix it on-the-fly or back it out depending.
If switching to Starman goes well, after a week or two, I will probably go ahead with moving the site to UTF-8 fully. The HTML has a way to indicate UTF-8. The database has a way to convert tables to utf8mb4 format (currently I'm using latin1 [single byte]). Perl (what most of the site is written in) has its way to switch to UTF8 as well.
#1 bill
I'm planning to update/swap-in an alternative piece of software that GameTZ uses to serve web pages.
Hopefully, it will just work and no one will even notice.
But, there may be issues. So, let me know if you notice things not working or seeming off.
I'm not sure when I'll do this, but perhaps off-and-on over the next few days, perhaps starting soon.
For people interested in the technical details:
For 20+ years GameTZ has used FCGI the Perl module that implements FastCGI. But, after the trouble last week with quotes getting turned into ?s because of character set issues, I realized FCGI does not support multibyte characters, and thus is essentially obsolete.
I found something newer (only 10 years old!) called PSGI that supports UTF-8 character sets (e.g multibyte characters). PSGI can basically be swapped in for FastCGI (using an emulation module).
I have been testing a PSGI-based server component and it seems to work well (on my test server). The component is called Starman (which is unfortunate because I've had that David Bowie song in my head all week because of it).
Once I put Starman on the live site, it will get a lot more usage and be truly tested. I may have to fix it on-the-fly or back it out depending.
If switching to Starman goes well, after a week or two, I will probably go ahead with moving the site to UTF-8 fully. The HTML has a way to indicate UTF-8. The database has a way to convert tables to utf8mb4 format (currently I'm using latin1 [single byte]). Perl (what most of the site is written in) has its way to switch to UTF8 as well.
26-Jan-23 7:08am
#5 benstylus
I just got a 503 error trying to click on this topic.
It worked when I clicked back and tried again though
#5 benstylus
I just got a 503 error trying to click on this topic.
It worked when I clicked back and tried again though
26-Jan-23 7:14am
#6 bill
Yeah, I'm working on something. So, I'm restarting the server a lot to test/fix stuff.
#6 bill
Yeah, I'm working on something. So, I'm restarting the server a lot to test/fix stuff.
26-Jan-23 9:41am
#7 bill
I've gone back to the old setup for now.
There's an issue where I'm not getting the IP address for the request correctly.
I'll work on correcting this on my test server ...
#7 bill
I've gone back to the old setup for now.
There's an issue where I'm not getting the IP address for the request correctly.
I'll work on correcting this on my test server ...
26-Jan-23 1:09pm
#10 bill
You're welcome, though I feel like I'm partly causing issue then fixing them. But, ultimately GameTZ should be in a better position technology-wise.
Note that I am still making some adjustments, so the site may be down for brief moments.
#10 bill
You're welcome, though I feel like I'm partly causing issue then fixing them. But, ultimately GameTZ should be in a better position technology-wise.
Note that I am still making some adjustments, so the site may be down for brief moments.
1-Feb-23 9:41pm
#12 nonamesleft
willyum wrote:
Thanks for all your hard work Bill!
This. Yes. Thank you Bill. This site is one of my favorites. Thanks for all you do for it.
#12 nonamesleft
willyum wrote:
Thanks for all your hard work Bill!
This. Yes. Thank you Bill. This site is one of my favorites. Thanks for all you do for it.2-Feb-23 8:26am
#13 bill
Thanks for the kind words.
More details for the curious
And, just a little update. I'm still trying to understand & fix the issue with using the better character set (UTF-8). I have it set up on my test server. But, it's still not right (garbled characters despite everything being set to use utf-8) and I don't know why. Character sets and how they react to computer languages are amazingly complicated. So, I'm not sure if I'll get there, but I'll keep trying things. The current live site is working well (though, I don't know how/why?!), so if I have to leave it like that, it's probably OK. But yeah, we should be using UTF-8...
Also, while looking into all this, I've come to realize that the operating system the site is using (Centos 7) is not great. Well, I knew it was being phased out, but that doesn't happen until 2024. So, I was going to let that play out. But, I see now that it's also rather old and doesn't have the latest versions of a lot of software the site uses. So, I should think about upgrading the site's OS sooner. And, it looks like Rocky Linux is the way to go for that. So, that's another project to work toward over the next few months...
#13 bill
Thanks for the kind words.
More details for the curious
And, just a little update. I'm still trying to understand & fix the issue with using the better character set (UTF-8). I have it set up on my test server. But, it's still not right (garbled characters despite everything being set to use utf-8) and I don't know why. Character sets and how they react to computer languages are amazingly complicated. So, I'm not sure if I'll get there, but I'll keep trying things. The current live site is working well (though, I don't know how/why?!), so if I have to leave it like that, it's probably OK. But yeah, we should be using UTF-8...
Also, while looking into all this, I've come to realize that the operating system the site is using (Centos 7) is not great. Well, I knew it was being phased out, but that doesn't happen until 2024. So, I was going to let that play out. But, I see now that it's also rather old and doesn't have the latest versions of a lot of software the site uses. So, I should think about upgrading the site's OS sooner. And, it looks like Rocky Linux is the way to go for that. So, that's another project to work toward over the next few months...
2-Feb-23 1:49pm
#14 nonamesleft
bill wrote:
Thanks for the kind words. More details for the curious ...Will UTF-8 address what happens when a user types a word using a special character on the mobile app?
It causes strange things to happen. Here's an example:
The word Pokemon, if you type it with that line on top of the e. Here's the result: Pokémon.
I didn't type any of those additional characters.
#14 nonamesleft
bill wrote:
Thanks for the kind words. More details for the curious ...Will UTF-8 address what happens when a user types a word using a special character on the mobile app?
It causes strange things to happen. Here's an example:
The word Pokemon, if you type it with that line on top of the e. Here's the result: Pokémon.
I didn't type any of those additional characters.
2-Feb-23 2:21pm
#15 bill
That's similar to the problems I see when I try to adjust the website to use UTF-8. The app uses similar technology to the website. It runs using a sort of locked-down web-browser module that tries to mimic a native app but is really just HTML and javascript.
I hadn't considered the app in all this. Honestly, it's not a priority. I'll add the app to my list of things to get right... with some luck it may just work better if I can fix the site to do utf-8.
I bet the app is forcing UTF-8 and thus runs into the problems I see when I try to do that on my test server with the website. e.g. If I set the html page encoding to utf-8 é becomes é (like your Pokemon example). But, I have other problems too.
I think there's something for me to learn from this. But, I haven't quite put it together in my head. All of these special characters (é Ã ©) are in the current character set of the site (ISO-8859-1). UTF-8 has them too, but I think with a different code. So, I think the trouble is coming from some layer of the software seeing the special characters and transforming them when it really shouldn't be doing that. I think if I could somehow stop that translation, I might fix it. But, I'm not sure where it's happening.
The live site (not the app, uses ISO-8859-1) either just works because it avoids these incorrect translations. Or, it's actually in a state where the translations do the right thing? I think there are cases when special characters get translated twice (and thus mangled).
One of the tricky things about this is all the layers, so it also matters what software I'm using to view/debug the broken software as it will also support UTF8 or not or be translating it. So, my terminal window, editor, programming language, database ... all have their own way of dealing with it.. of course, but it's like the fog of war.
#15 bill
That's similar to the problems I see when I try to adjust the website to use UTF-8. The app uses similar technology to the website. It runs using a sort of locked-down web-browser module that tries to mimic a native app but is really just HTML and javascript.
I hadn't considered the app in all this. Honestly, it's not a priority. I'll add the app to my list of things to get right... with some luck it may just work better if I can fix the site to do utf-8.
I bet the app is forcing UTF-8 and thus runs into the problems I see when I try to do that on my test server with the website. e.g. If I set the html page encoding to utf-8 é becomes é (like your Pokemon example). But, I have other problems too.
I think there's something for me to learn from this. But, I haven't quite put it together in my head. All of these special characters (é Ã ©) are in the current character set of the site (ISO-8859-1). UTF-8 has them too, but I think with a different code. So, I think the trouble is coming from some layer of the software seeing the special characters and transforming them when it really shouldn't be doing that. I think if I could somehow stop that translation, I might fix it. But, I'm not sure where it's happening.
The live site (not the app, uses ISO-8859-1) either just works because it avoids these incorrect translations. Or, it's actually in a state where the translations do the right thing? I think there are cases when special characters get translated twice (and thus mangled).
One of the tricky things about this is all the layers, so it also matters what software I'm using to view/debug the broken software as it will also support UTF8 or not or be translating it. So, my terminal window, editor, programming language, database ... all have their own way of dealing with it.. of course, but it's like the fog of war.
2-Feb-23 3:27pm
#16 nonamesleft
Sounds pretty tricky, figuring that out. It's not really an issue, the character swaps. I was just wondering if changing the character set would fix it, that's all.
Another strange thing that happens, is that if someone replies to a post that has a special character in it, the original word gets elongated with more symbols, inside the greyed replying box.
For instance, here's the word with that e, Pokémon. I'm going to reply to this post and you'll see what happens to the word in the greyed box.
#16 nonamesleft
Sounds pretty tricky, figuring that out. It's not really an issue, the character swaps. I was just wondering if changing the character set would fix it, that's all.
Another strange thing that happens, is that if someone replies to a post that has a special character in it, the original word gets elongated with more symbols, inside the greyed replying box.
For instance, here's the word with that e, Pokémon. I'm going to reply to this post and you'll see what happens to the word in the greyed box.
2-Feb-23 3:28pm
#17 nonamesleft
nonamesleft wrote:
Sounds pretty tricky, figuring that out. It's not really an issue, the character swaps. I was just wondering if changing the character set would fix it, that's all. Another strange thing that happens, is that if someone replies to a post that has a special character in it, the original word gets elongated with more symbols, inside the greyed replying box. For instance, here's the word with that e, Pokémon. I'm going to reply to this post and you'll see what happens to the word in the greyed box.Compare the above Pokemon spelling with the previous post.
#17 nonamesleft
nonamesleft wrote:
Sounds pretty tricky, figuring that out. It's not really an issue, the character swaps. I was just wondering if changing the character set would fix it, that's all. Another strange thing that happens, is that if someone replies to a post that has a special character in it, the original word gets elongated with more symbols, inside the greyed replying box. For instance, here's the word with that e, Pokémon. I'm going to reply to this post and you'll see what happens to the word in the greyed box.Compare the above Pokemon spelling with the previous post.
2-Feb-23 4:15pm
#18 bill
Yeah, interesting. I think it's probably re-affirming that there's some sort of translation that is done to the Pokemon and again with the result such that it grows in length.
An aspect of these character sets is that UTF-8 supports "multi-byte" characters while the current site uses ISO-8859-1 which is single byte characters. So, the accented e characters becoming A with a squiggle over it and a copyright symbol is likely a single-byte characters being translated into a 2-byte character which then gets treated as 2 characters by ISO-8859-1 because it thinks each byte is a different character. ...if that makes sense. Then, when you reply to it, the 2 double again becoming 4 or perhaps more. UTF-8 supports 4-byte characters too.
But yeah, none of that should happen. The software at all levels should understand it's all UTF-8 and not garble it ever.
#18 bill
Yeah, interesting. I think it's probably re-affirming that there's some sort of translation that is done to the Pokemon and again with the result such that it grows in length.
An aspect of these character sets is that UTF-8 supports "multi-byte" characters while the current site uses ISO-8859-1 which is single byte characters. So, the accented e characters becoming A with a squiggle over it and a copyright symbol is likely a single-byte characters being translated into a 2-byte character which then gets treated as 2 characters by ISO-8859-1 because it thinks each byte is a different character. ...if that makes sense. Then, when you reply to it, the 2 double again becoming 4 or perhaps more. UTF-8 supports 4-byte characters too.
But yeah, none of that should happen. The software at all levels should understand it's all UTF-8 and not garble it ever.
2-Feb-23 4:43pm
#19 bill
So yeah, trying to look up encoding for the accented e in Pokemon (é)
https://en.wikipedia... says it's the 2 bytes 195 169 in UTF-8.
If I look up 195 in ISO-8859-1 it is "capital a, tilde" (Ã) and 169 is copyright (©). Thus Pokémon becomes Pokémon .
So, maybe it's something like your phone via the gtz app is sending a UTF-8 string of bytes for Pokemon that has 2 bytes for the accented e. But, my site is treating it like each byte is a different character and thus the A tilde and copyright characters come out of that.
#19 bill
So yeah, trying to look up encoding for the accented e in Pokemon (é)
https://en.wikipedia... says it's the 2 bytes 195 169 in UTF-8.
If I look up 195 in ISO-8859-1 it is "capital a, tilde" (Ã) and 169 is copyright (©). Thus Pokémon becomes Pokémon .
So, maybe it's something like your phone via the gtz app is sending a UTF-8 string of bytes for Pokemon that has 2 bytes for the accented e. But, my site is treating it like each byte is a different character and thus the A tilde and copyright characters come out of that.
2-Feb-23 5:42pm
#20 nonamesleft
@Bill Thanks for looking into this. It's interesting stuff.
I wonder if it's what you're saying and that it's just via the app. I'm going to try posting that same word via the classic site, to see if the same thing happens.
While I'm at it, I'll try several other special characters, just to see if there's a different result.
ä½ å¥½
ã“ã‚“ã«ã¡ã¯
jalapeño
Ë
Ë
É
é
Å’
ø
#20 nonamesleft
@Bill Thanks for looking into this. It's interesting stuff.
I wonder if it's what you're saying and that it's just via the app. I'm going to try posting that same word via the classic site, to see if the same thing happens.
While I'm at it, I'll try several other special characters, just to see if there's a different result.
ä½ å¥½
ã“ã‚“ã«ã¡ã¯
jalapeño
Ë
Ë
É
é
Å’
ø
Updating GameTZ's software