quote, apostrophe, special characters are ?s
17-Jan-23 11:44am
#1 MrBean
Anyone else seeing this? Mobile and desktop, many (not all), single quotes are being converted to question marks.
#1 MrBean
Anyone else seeing this? Mobile and desktop, many (not all), single quotes are being converted to question marks.
17-Jan-23 12:35pm
#2 bill
hm... Yes, someone else mentioned this yesterday. I had just updated the database software. I didn't think it could cause this, but maybe it has.
I believe it's related to character encoding. There's a standard set of characters (letters, numbers, punctuation) that is called ASCII. It's old and works well, but lacks variety and many less common characters. The other person who reported this had used and ellipses (...) characters (rather than 3 periods) and it got messed up (saw a ? too). I think the ? shows when there's some character encoding issue.
Quotes have the same potential problem. ASCII has single and double quotes, but they are straight and lack the curved variations (true punctuation has the quotes curving differently on either side of the words they contain). So, some modern operating systems (mobile devices) use the non-ASCII characters now. Those are the ones that can end up being ?'s.
I think the database may be involved as it may not have the right character encoding to store them ... or something like that.
I'll try to look into it... maybe take some time.
A workaround is to try to mostly use ASCII characters as much as possible.
#2 bill
hm... Yes, someone else mentioned this yesterday. I had just updated the database software. I didn't think it could cause this, but maybe it has.
I believe it's related to character encoding. There's a standard set of characters (letters, numbers, punctuation) that is called ASCII. It's old and works well, but lacks variety and many less common characters. The other person who reported this had used and ellipses (...) characters (rather than 3 periods) and it got messed up (saw a ? too). I think the ? shows when there's some character encoding issue.
Quotes have the same potential problem. ASCII has single and double quotes, but they are straight and lack the curved variations (true punctuation has the quotes curving differently on either side of the words they contain). So, some modern operating systems (mobile devices) use the non-ASCII characters now. Those are the ones that can end up being ?'s.
I think the database may be involved as it may not have the right character encoding to store them ... or something like that.
I'll try to look into it... maybe take some time.
A workaround is to try to mostly use ASCII characters as much as possible.
17-Jan-23 12:36pm
#3 bill
@dunno001 I think this is what you had mentioned yesterday with the ellipses issue you are having.
I think you may be right, that it's the database upgrade that caused this too.
#3 bill
@dunno001 I think this is what you had mentioned yesterday with the ellipses issue you are having.
I think you may be right, that it's the database upgrade that caused this too.
17-Jan-23 12:51pm
#5 Slickriven
Am seeing this in other people's posts, expecting it'll change the 2 single quotes here as well. Interesting issue to deal with Bill. Surprised you're not using Unicode vs. ASCII.
#5 Slickriven
Am seeing this in other people's posts, expecting it'll change the 2 single quotes here as well. Interesting issue to deal with Bill. Surprised you're not using Unicode vs. ASCII.
17-Jan-23 1:16pm
#9 bill
bill wrote:
?Test?. ? ?quote?
I used my phone to do those quotes... and, as expected it used the special curved quote characters.
Alternately, using a laptop.
"test" ... 'quote'
The old-style straight quotes (ASCII) and I assume these still look fine.
#9 bill
bill wrote:
?Test?. ? ?quote?
I used my phone to do those quotes... and, as expected it used the special curved quote characters.
Alternately, using a laptop.
"test" ... 'quote'
The old-style straight quotes (ASCII) and I assume these still look fine.
17-Jan-23 1:22pm
#10 bill
I'm looking at the database stuff and not finding what might have changed.
Also, that it's even more complicated as they call the character encoding I use "latin1". Which is not just ASCII (has a lot of other special accented characters and such). So, I think it would still be fine.
There's also a layer in my own code that trying to "fix" some of the common stuff... maybe that somehow broke. But, it's hard to tell what upgrading the DB software changed.
And, there's some danger here if I try converting to another character set encoding... I could trash all the data. Hopefully, that's unlikely.
I've haven't used UTF-8 (now synonymous utf8mb3 in the database world apparently). The "latin1" encoding I use has some advantages as it uses just one byte per character. It's often faster because of that. I guess if I just switch to UTF-8, the database could become quite slow in seemingly random ways. I think since we're 99% English, latin1 is more than enough. So, sticking with that seems like a good idea still. I'm still reading about it, though.
#10 bill
I'm looking at the database stuff and not finding what might have changed.
Also, that it's even more complicated as they call the character encoding I use "latin1". Which is not just ASCII (has a lot of other special accented characters and such). So, I think it would still be fine.
There's also a layer in my own code that trying to "fix" some of the common stuff... maybe that somehow broke. But, it's hard to tell what upgrading the DB software changed.
And, there's some danger here if I try converting to another character set encoding... I could trash all the data. Hopefully, that's unlikely.
I've haven't used UTF-8 (now synonymous utf8mb3 in the database world apparently). The "latin1" encoding I use has some advantages as it uses just one byte per character. It's often faster because of that. I guess if I just switch to UTF-8, the database could become quite slow in seemingly random ways. I think since we're 99% English, latin1 is more than enough. So, sticking with that seems like a good idea still. I'm still reading about it, though.
17-Jan-23 3:25pm
#11 bill
Well, I haven't tracked down what changed or why, but my best guess is the database software is now (since updating) more strict about what it allows to be stored. So, it used to be that people would send special characters that were not in the default "latin1" character set and it would save them in the database regardless. Now, the characters that aren't in "latin1" are changed into ?.
What I can do is convert such characters to the blander variants that are in "latin1", so it will work. But, this will mean finding all the places where stuff people type gets saved in the database (e.g. forum posts, pms, offers trades, bio, ...) There are a lot of places, so that may take a while and I could still miss some. Still, it could allow for other ways to fix inputs, which could be useful in other cases as well.
I should keep looking for the cause, though. There may be some way to just do this automatically.
#11 bill
Well, I haven't tracked down what changed or why, but my best guess is the database software is now (since updating) more strict about what it allows to be stored. So, it used to be that people would send special characters that were not in the default "latin1" character set and it would save them in the database regardless. Now, the characters that aren't in "latin1" are changed into ?.
What I can do is convert such characters to the blander variants that are in "latin1", so it will work. But, this will mean finding all the places where stuff people type gets saved in the database (e.g. forum posts, pms, offers trades, bio, ...) There are a lot of places, so that may take a while and I could still miss some. Still, it could allow for other ways to fix inputs, which could be useful in other cases as well.
I should keep looking for the cause, though. There may be some way to just do this automatically.
17-Jan-23 4:56pm
#12 DarkFact
I always had issues with this crap on Android. Think I made a topic about it back in the day too, maybe. At some point you tightened the screws somewhere and it took care of it. Actually I think my issue may have been with apostrophes turning into & or something thinking on it again
#12 DarkFact
I always had issues with this crap on Android. Think I made a topic about it back in the day too, maybe. At some point you tightened the screws somewhere and it took care of it. Actually I think my issue may have been with apostrophes turning into & or something thinking on it again
17-Jan-23 5:20pm
#13 Alaisiagae
Yes, I noticed the other day than an o with an umlaut, which showed normally before the database update, was changed initially to some gobbly gook (some random characters). I tried re-entering it, and when I saved, it changed to a question mark.
#13 Alaisiagae
Yes, I noticed the other day than an o with an umlaut, which showed normally before the database update, was changed initially to some gobbly gook (some random characters). I tried re-entering it, and when I saved, it changed to a question mark.
17-Jan-23 8:05pm
#14 MrBean
bill wrote:
bill wrote:> ?Test?. ? ?quote? I used my phone to do those quotes... and, as expected it used the special curved quote characters. Alternately, using a laptop. "test" ... 'quote' The old-style straight quotes (ASCII) and I assume these still look fine.
This looks good here
#14 MrBean
bill wrote:
bill wrote:> ?Test?. ? ?quote? I used my phone to do those quotes... and, as expected it used the special curved quote characters. Alternately, using a laptop. "test" ... 'quote' The old-style straight quotes (ASCII) and I assume these still look fine.
This looks good here
17-Jan-23 10:44pm
#17 MrBean
WithinTemptation wrote:
Pok?mon
I read this as a Jamaican asking me if I wanted some Pok, Mon!
#17 MrBean
WithinTemptation wrote:
Pok?mon
I read this as a Jamaican asking me if I wanted some Pok, Mon!
18-Jan-23 4:45am
#18 bill
I feel like this is GameTZ showing its age. It still uses this old character set that used to be what everyone used (20 years ago). Now, not so much. But, I'm not sure I can just magically move to the new normal without the site being shredded in the process because it was all built on that old character set. Losing sleep over this... RIP GameTZ.
#18 bill
I feel like this is GameTZ showing its age. It still uses this old character set that used to be what everyone used (20 years ago). Now, not so much. But, I'm not sure I can just magically move to the new normal without the site being shredded in the process because it was all built on that old character set. Losing sleep over this... RIP GameTZ.
18-Jan-23 9:29am
#24 bill
"Good morning, Dave," said HAL.
'Good morning, Dave,' said HAL.
"Good morning, Dave," said HAL.
'Good morning, Dave,' said HAL.
#24 bill
"Good morning, Dave," said HAL.
'Good morning, Dave,' said HAL.
"Good morning, Dave," said HAL.
'Good morning, Dave,' said HAL.
18-Jan-23 9:47am
#27 bill
Maybe, it's fixed now.
Let me know if you still have problems.
Some notes to my future self (and the tech-curious)...
I'm still not sure what caused it to break exactly, but there are some variables the DB uses to set character set. I think they had been latin1 by default, but for the web code I have, (after the update perhaps - can't be sure) it was utf8mb4 and that caused issues (the ? we were seeing) because I'm using latin1 in my db and it turns utf chars into ?s. So, I now set the character set before using the database from my web code to latin1 and that seems to make it work like it used to.
There is still this nagging potential issue that GameTZ is using the latin1 character set despite HTML5 standards and the database people clearly pushing UTF-8. I did try to switch to UTF-8 on my test server (converted a couple tables, set the html page charset=UTF-8) and it mostly works, but I saw an issue. When I searched the forum for Pokémon, the results pages had Pokémon [garbage characters in the search text box where the accented e should have been). Switching to UTF-8 is a fair bit of work too, so it seems best to hold off on that indefinitely give the risk/reward being dubious at this point.
#27 bill
Maybe, it's fixed now.
Let me know if you still have problems.
Some notes to my future self (and the tech-curious)...
I'm still not sure what caused it to break exactly, but there are some variables the DB uses to set character set. I think they had been latin1 by default, but for the web code I have, (after the update perhaps - can't be sure) it was utf8mb4 and that caused issues (the ? we were seeing) because I'm using latin1 in my db and it turns utf chars into ?s. So, I now set the character set before using the database from my web code to latin1 and that seems to make it work like it used to.
There is still this nagging potential issue that GameTZ is using the latin1 character set despite HTML5 standards and the database people clearly pushing UTF-8. I did try to switch to UTF-8 on my test server (converted a couple tables, set the html page charset=UTF-8) and it mostly works, but I saw an issue. When I searched the forum for Pokémon, the results pages had Pokémon [garbage characters in the search text box where the accented e should have been). Switching to UTF-8 is a fair bit of work too, so it seems best to hold off on that indefinitely give the risk/reward being dubious at this point.
18-Jan-23 9:48am
#28 bill
I should also note that all the bad ?s in entered text for the last couple days stored the ?s, so it's still like that. I don't have any great way to fix all those. So, I was just going to not fix those. From now on, it should store them correctly, though.
#28 bill
I should also note that all the bad ?s in entered text for the last couple days stored the ?s, so it's still like that. I don't have any great way to fix all those. So, I was just going to not fix those. From now on, it should store them correctly, though.
18-Jan-23 1:17pm
#39 bill
Good.
So, it seems fairly stable for now, but I feel more and more like I should move the whole site to UTF-8.
I'll spend more time playing with it on my test server. If it seems good, I may push it out to the live site. We'll see.
#39 bill
Good.
So, it seems fairly stable for now, but I feel more and more like I should move the whole site to UTF-8.
I'll spend more time playing with it on my test server. If it seems good, I may push it out to the live site. We'll see.
18-Jan-23 5:20pm
#43 John
I̷̠̞̘͕̞̓̐̽̔̄ͦ͂͡ ̸̨̲̜̃̇͂̑͜å͚͕͓̮̘ͥ̔͒ͪͥ͠m̮͍͕̭̍̃ͥ͆̾̋͆ͤ ͙̪̖͓̫͎͔̘̬̾ͭ̄ͥ́̂̔́Ğ̞̠̠̲̯̪̳͈̄̇͐̒̆̍́o̵̟ͮ̍̐ͭẓ̈̚ě͓̬̬̯̘̓ͪ̇̊ͪͮř̸̛̭̬͈͉̩̗̺͈̱ͮ͟,̲͇͚̌͐̕ ͐̇̾̔̔͒̾͠͏̙̫̪͜D͎ͦ̈̂͒̈̚͢ę͇̥͐͋̌ͣ́͜s̗̣̫̟̞̰͂̋̈́̄ͅţ̡̣̯͊ͭ͆̔̚ͅr̷̜̤̎͂̑ͯ̓ͮ̆̀ő̿͒͒́̾ͭ̀̚҉͇̮̮̺̞͎͖ͅy͓̺̰̖̠̋̈͗̃̈́̀̓̋̀̚ẽ̹̋̑͜r̗͍̗̼͔͙̘ͨ͛̀͘͝ ̨̝̗̼͔͋͞ͅo̸̷̮̞̣̜̺̮̪̊̉͠f̨ͣͥ̔͋͗͛̉̚҉̷̦̳̪̳ ̝ͤ̀͝Ŵ̛̞̺̈́ͦ͜ǒͦ̑̌̉̾̿̚͏̜̹͎͚̺̳r̴̟̳̂̓͊̓͂̀̎͠l̴̵͎̞̈́̿͗ͦd̡̜̬͔̹ͭ͌̽̆̔͒ͣͧs̀͂̇͂ͮ̄̊͘҉̥̤̱͎.̠ͤ̒̐̓ͥ
#43 John
I̷̠̞̘͕̞̓̐̽̔̄ͦ͂͡ ̸̨̲̜̃̇͂̑͜å͚͕͓̮̘ͥ̔͒ͪͥ͠m̮͍͕̭̍̃ͥ͆̾̋͆ͤ ͙̪̖͓̫͎͔̘̬̾ͭ̄ͥ́̂̔́Ğ̞̠̠̲̯̪̳͈̄̇͐̒̆̍́o̵̟ͮ̍̐ͭẓ̈̚ě͓̬̬̯̘̓ͪ̇̊ͪͮř̸̛̭̬͈͉̩̗̺͈̱ͮ͟,̲͇͚̌͐̕ ͐̇̾̔̔͒̾͠͏̙̫̪͜D͎ͦ̈̂͒̈̚͢ę͇̥͐͋̌ͣ́͜s̗̣̫̟̞̰͂̋̈́̄ͅţ̡̣̯͊ͭ͆̔̚ͅr̷̜̤̎͂̑ͯ̓ͮ̆̀ő̿͒͒́̾ͭ̀̚҉͇̮̮̺̞͎͖ͅy͓̺̰̖̠̋̈͗̃̈́̀̓̋̀̚ẽ̹̋̑͜r̗͍̗̼͔͙̘ͨ͛̀͘͝ ̨̝̗̼͔͋͞ͅo̸̷̮̞̣̜̺̮̪̊̉͠f̨ͣͥ̔͋͗͛̉̚҉̷̦̳̪̳ ̝ͤ̀͝Ŵ̛̞̺̈́ͦ͜ǒͦ̑̌̉̾̿̚͏̜̹͎͚̺̳r̴̟̳̂̓͊̓͂̀̎͠l̴̵͎̞̈́̿͗ͦd̡̜̬͔̹ͭ͌̽̆̔͒ͣͧs̀͂̇͂ͮ̄̊͘҉̥̤̱͎.̠ͤ̒̐̓ͥ
18-Jan-23 7:05pm
#44 Kommie
DarkFact wrote:
{imgt}https://media.discor... 750 1038{imgt} Uh oh. This is new. Edge browser on iPhone
Why are you using Edge? All the cool geezers use Internet Explorer.
#44 Kommie
DarkFact wrote:
{imgt}https://media.discor... 750 1038{imgt} Uh oh. This is new. Edge browser on iPhone
Why are you using Edge? All the cool geezers use Internet Explorer.
18-Jan-23 10:06pm
#45 MrBean
Kommie wrote:
DarkFact wrote:> {imgt}https://media.discor...> 750 1038{imgt}> > Uh oh. This is new. Edge browser on iPhone Why are you using Edge? All the cool geezers use Internet Explorer.
Edge is actually quite awesome. Also IE will be officially removed next month, finally!
#45 MrBean
Kommie wrote:
DarkFact wrote:> {imgt}https://media.discor...> 750 1038{imgt}> > Uh oh. This is new. Edge browser on iPhone Why are you using Edge? All the cool geezers use Internet Explorer.
Edge is actually quite awesome. Also IE will be officially removed next month, finally!
quote, apostrophe, special characters are ?s