quote, apostrophe, special characters are ?s
17-Jan-23 11:44am
#1 MrBean
Anyone else seeing this? Mobile and desktop, many (not all), single quotes are being converted to question marks.
#1 MrBean
Anyone else seeing this? Mobile and desktop, many (not all), single quotes are being converted to question marks.
17-Jan-23 12:35pm
#2 bill
hm... Yes, someone else mentioned this yesterday. I had just updated the database software. I didn't think it could cause this, but maybe it has.
I believe it's related to character encoding. There's a standard set of characters (letters, numbers, punctuation) that is called ASCII. It's old and works well, but lacks variety and many less common characters. The other person who reported this had used and ellipses (...) characters (rather than 3 periods) and it got messed up (saw a ? too). I think the ? shows when there's some character encoding issue.
Quotes have the same potential problem. ASCII has single and double quotes, but they are straight and lack the curved variations (true punctuation has the quotes curving differently on either side of the words they contain). So, some modern operating systems (mobile devices) use the non-ASCII characters now. Those are the ones that can end up being ?'s.
I think the database may be involved as it may not have the right character encoding to store them ... or something like that.
I'll try to look into it... maybe take some time.
A workaround is to try to mostly use ASCII characters as much as possible.
#2 bill
hm... Yes, someone else mentioned this yesterday. I had just updated the database software. I didn't think it could cause this, but maybe it has.
I believe it's related to character encoding. There's a standard set of characters (letters, numbers, punctuation) that is called ASCII. It's old and works well, but lacks variety and many less common characters. The other person who reported this had used and ellipses (...) characters (rather than 3 periods) and it got messed up (saw a ? too). I think the ? shows when there's some character encoding issue.
Quotes have the same potential problem. ASCII has single and double quotes, but they are straight and lack the curved variations (true punctuation has the quotes curving differently on either side of the words they contain). So, some modern operating systems (mobile devices) use the non-ASCII characters now. Those are the ones that can end up being ?'s.
I think the database may be involved as it may not have the right character encoding to store them ... or something like that.
I'll try to look into it... maybe take some time.
A workaround is to try to mostly use ASCII characters as much as possible.
17-Jan-23 12:36pm
#3 bill
@dunno001 I think this is what you had mentioned yesterday with the ellipses issue you are having.
I think you may be right, that it's the database upgrade that caused this too.
#3 bill
@dunno001 I think this is what you had mentioned yesterday with the ellipses issue you are having.
I think you may be right, that it's the database upgrade that caused this too.
17-Jan-23 12:51pm
#5 Slickriven
Am seeing this in other people's posts, expecting it'll change the 2 single quotes here as well. Interesting issue to deal with Bill. Surprised you're not using Unicode vs. ASCII.
#5 Slickriven
Am seeing this in other people's posts, expecting it'll change the 2 single quotes here as well. Interesting issue to deal with Bill. Surprised you're not using Unicode vs. ASCII.
17-Jan-23 1:16pm
#9 bill
bill wrote:
?Test?. ? ?quote?
I used my phone to do those quotes... and, as expected it used the special curved quote characters.
Alternately, using a laptop.
"test" ... 'quote'
The old-style straight quotes (ASCII) and I assume these still look fine.
#9 bill
bill wrote:
?Test?. ? ?quote?
I used my phone to do those quotes... and, as expected it used the special curved quote characters.
Alternately, using a laptop.
"test" ... 'quote'
The old-style straight quotes (ASCII) and I assume these still look fine.
17-Jan-23 1:22pm
#10 bill
I'm looking at the database stuff and not finding what might have changed.
Also, that it's even more complicated as they call the character encoding I use "latin1". Which is not just ASCII (has a lot of other special accented characters and such). So, I think it would still be fine.
There's also a layer in my own code that trying to "fix" some of the common stuff... maybe that somehow broke. But, it's hard to tell what upgrading the DB software changed.
And, there's some danger here if I try converting to another character set encoding... I could trash all the data. Hopefully, that's unlikely.
I've haven't used UTF-8 (now synonymous utf8mb3 in the database world apparently). The "latin1" encoding I use has some advantages as it uses just one byte per character. It's often faster because of that. I guess if I just switch to UTF-8, the database could become quite slow in seemingly random ways. I think since we're 99% English, latin1 is more than enough. So, sticking with that seems like a good idea still. I'm still reading about it, though.
#10 bill
I'm looking at the database stuff and not finding what might have changed.
Also, that it's even more complicated as they call the character encoding I use "latin1". Which is not just ASCII (has a lot of other special accented characters and such). So, I think it would still be fine.
There's also a layer in my own code that trying to "fix" some of the common stuff... maybe that somehow broke. But, it's hard to tell what upgrading the DB software changed.
And, there's some danger here if I try converting to another character set encoding... I could trash all the data. Hopefully, that's unlikely.
I've haven't used UTF-8 (now synonymous utf8mb3 in the database world apparently). The "latin1" encoding I use has some advantages as it uses just one byte per character. It's often faster because of that. I guess if I just switch to UTF-8, the database could become quite slow in seemingly random ways. I think since we're 99% English, latin1 is more than enough. So, sticking with that seems like a good idea still. I'm still reading about it, though.
17-Jan-23 3:25pm
#11 bill
Well, I haven't tracked down what changed or why, but my best guess is the database software is now (since updating) more strict about what it allows to be stored. So, it used to be that people would send special characters that were not in the default "latin1" character set and it would save them in the database regardless. Now, the characters that aren't in "latin1" are changed into ?.
What I can do is convert such characters to the blander variants that are in "latin1", so it will work. But, this will mean finding all the places where stuff people type gets saved in the database (e.g. forum posts, pms, offers trades, bio, ...) There are a lot of places, so that may take a while and I could still miss some. Still, it could allow for other ways to fix inputs, which could be useful in other cases as well.
I should keep looking for the cause, though. There may be some way to just do this automatically.
#11 bill
Well, I haven't tracked down what changed or why, but my best guess is the database software is now (since updating) more strict about what it allows to be stored. So, it used to be that people would send special characters that were not in the default "latin1" character set and it would save them in the database regardless. Now, the characters that aren't in "latin1" are changed into ?.
What I can do is convert such characters to the blander variants that are in "latin1", so it will work. But, this will mean finding all the places where stuff people type gets saved in the database (e.g. forum posts, pms, offers trades, bio, ...) There are a lot of places, so that may take a while and I could still miss some. Still, it could allow for other ways to fix inputs, which could be useful in other cases as well.
I should keep looking for the cause, though. There may be some way to just do this automatically.
17-Jan-23 4:56pm
#12 DarkFact
I always had issues with this crap on Android. Think I made a topic about it back in the day too, maybe. At some point you tightened the screws somewhere and it took care of it. Actually I think my issue may have been with apostrophes turning into & or something thinking on it again
#12 DarkFact
I always had issues with this crap on Android. Think I made a topic about it back in the day too, maybe. At some point you tightened the screws somewhere and it took care of it. Actually I think my issue may have been with apostrophes turning into & or something thinking on it again
17-Jan-23 5:20pm
#13 Alaisiagae
Yes, I noticed the other day than an o with an umlaut, which showed normally before the database update, was changed initially to some gobbly gook (some random characters). I tried re-entering it, and when I saved, it changed to a question mark.
#13 Alaisiagae
Yes, I noticed the other day than an o with an umlaut, which showed normally before the database update, was changed initially to some gobbly gook (some random characters). I tried re-entering it, and when I saved, it changed to a question mark.
17-Jan-23 8:05pm
#14 MrBean
bill wrote:
bill wrote:> ?Test?. ? ?quote? I used my phone to do those quotes... and, as expected it used the special curved quote characters. Alternately, using a laptop. "test" ... 'quote' The old-style straight quotes (ASCII) and I assume these still look fine.
This looks good here
#14 MrBean
bill wrote:
bill wrote:> ?Test?. ? ?quote? I used my phone to do those quotes... and, as expected it used the special curved quote characters. Alternately, using a laptop. "test" ... 'quote' The old-style straight quotes (ASCII) and I assume these still look fine.
This looks good here
17-Jan-23 10:44pm
#17 MrBean
WithinTemptation wrote:
Pok?mon
I read this as a Jamaican asking me if I wanted some Pok, Mon!
#17 MrBean
WithinTemptation wrote:
Pok?mon
I read this as a Jamaican asking me if I wanted some Pok, Mon!
18-Jan-23 4:45am
#18 bill
I feel like this is GameTZ showing its age. It still uses this old character set that used to be what everyone used (20 years ago). Now, not so much. But, I'm not sure I can just magically move to the new normal without the site being shredded in the process because it was all built on that old character set. Losing sleep over this... RIP GameTZ.
#18 bill
I feel like this is GameTZ showing its age. It still uses this old character set that used to be what everyone used (20 years ago). Now, not so much. But, I'm not sure I can just magically move to the new normal without the site being shredded in the process because it was all built on that old character set. Losing sleep over this... RIP GameTZ.
quote, apostrophe, special characters are ?s