Wrecked Games

Please login or register.

Login with username, password and session length
Advanced search  

News:

We're just that awesome.

Pages: [1] 2

Author Topic: Working on Unicode Support  (Read 3482 times)

pjcast

  • Administrator
  • Veteran
  • *****
  • Karma: +1/-0
  • Posts: 2661
    • View Profile
    • http://www.wreckedgames.com
Working on Unicode Support
« on: February 14, 2006, 03:18:11 PM »

I thought I'd be going with a Textlistener approach for handling Unicode, However,on third thought, i am going to try to use the existing KeyEvent to store a unicode charcter - so, on a keyDown event, you can check that KeyEvent element for the key's text translation (if any are present)... I will however, get something to work before I make a decision either way and commit anything to Cvs.
Logged

pjcast

  • Administrator
  • Veteran
  • *****
  • Karma: +1/-0
  • Posts: 2661
    • View Profile
    • http://www.wreckedgames.com
Working on Unicode Support
« Reply #1 on: February 15, 2006, 06:08:09 PM »

Linux build will be broken in Cvs head for a little spell while I am working on Unicode/Locale support. As always, for a stable branch, checkout the last release (0.4.0).

Initial Support for dead keys is being checked into Cvs now. Win32 should be functional.
Logged

pjcast

  • Administrator
  • Veteran
  • *****
  • Karma: +1/-0
  • Posts: 2661
    • View Profile
    • http://www.wreckedgames.com
Working on Unicode Support
« Reply #2 on: February 15, 2006, 07:56:50 PM »

Ok, I updated Linux to use XLookupString instead of the lookup table. I have no idea how this works for other languages or for deadkeys - but, I think I need to combine the output from it if the buffer length is greater than 1 char, so I will keep working on it. but at least it is not a hardcoded thing. Now, i just need to implement getAsString on Win32 and Linux to be almost there.
Logged

pjcast

  • Administrator
  • Veteran
  • *****
  • Karma: +1/-0
  • Posts: 2661
    • View Profile
    • http://www.wreckedgames.com
Working on Unicode Support
« Reply #3 on: February 16, 2006, 10:16:13 AM »

Ok dokey.. I just commited the getAsString() method of LinuxKeyboard to use XKeysymToString so that now the key name returned should work for any and all keyboard layouts/languages.

Now, just need to implement the getAsString on Win32Keyboard using the win32 API lookup key name functions.

To sum up what works right now:
 :arrow: Language Nuetral KeyCode to Character on Win32 - Even returns 5 French deadkey characters (need to work on adding all deakkey combinations).
 :arrow: Language Nuetral KeyCode to Character on Linux - No dead keys are handled at all.
 :arrow: getAsString returns layout/language nuetral key name on Linux
 :arrow: Win32 Defaults to NONEXCLUSIVE Keybaord access so that the translation works, else caps lock and shift have no effect.

To Do:
 :arrow: Implement getAsString on Win32 Keyboard
 :arrow: Try to handle deadkey combinations on Linux
 :arrow: Test all supported platforms against with multiple languages.
 :arrow: Get ready to release 0.5.0
Logged

pjcast

  • Administrator
  • Veteran
  • *****
  • Karma: +1/-0
  • Posts: 2661
    • View Profile
    • http://www.wreckedgames.com
Working on Unicode Support
« Reply #4 on: February 16, 2006, 12:49:52 PM »

I knocked out the getAsString on Win32.. Unfortunately, the Win32 API GetKeyNameText seems to crap out on extended keys (it returns no name). So, I just returned a Key_# in that case. This is unfortunate, and is kinda crappy as this covers many keys (Right ALT, Right CTRL, arrow keys, and some other NumPad keys).

I will keep an eye out for a work around, but the getAsString is not that important, it's only good use is for key mapping and display a key to the user. If a client app really needs to show that the key is an arrow key (or any of the extended keys) they can search the returned key string for a Key_ prefix and then look at the KeyCode themselves and name the key. As there are so many languages, i do not want to use a lookup table anymore in OIS.
Logged

cdsnyder

  • Regular
  • *
  • Karma: +0/-0
  • Posts: 26
    • View Profile
Working on Unicode Support
« Reply #5 on: February 18, 2006, 03:30:55 PM »

I was reading about Unicode for OSX. The provided means of getting Unicode for keycodes is a unicode string.  Their example was that certain indian keyboards can sometimes produce up to 3 unicode chars per single keycode, and other languages producing sometimes 2 chars per keycode. Maybe this should be reflected in the txt of a KeyEvent, which currently can only store one unicode char. I don't know how rare it is.. but apparently it does happen.
Logged

pjcast

  • Administrator
  • Veteran
  • *****
  • Karma: +1/-0
  • Posts: 2661
    • View Profile
    • http://www.wreckedgames.com
Working on Unicode Support
« Reply #6 on: February 18, 2006, 03:39:17 PM »

Well, currently, Linux and Win32 only really provide ascii not unicode (at least ascii on Win32). There is the tounicode method of Win32, but I think that just displaying Ascii at least in the native locale is a good starting point. I cannot throughly test implementing other languages or unicode as I am not entirely verses on unicode yet, and have a limited language base that I know (English).

Kinda wierd how One key would provide more then one unicode character, never heard that. I know a combination of keys can produce one character, never thought about the other way. I think that is too extreme a situation to handle, that would be better added as a patch from a native user of that language who experiences a case where they need that.
Logged

cdsnyder

  • Regular
  • *
  • Karma: +0/-0
  • Posts: 26
    • View Profile
Working on Unicode Support
« Reply #7 on: February 18, 2006, 05:02:35 PM »

Ok, np. Just letting ya know.
Logged

Rackle

  • Regular
  • *
  • Karma: +0/-0
  • Posts: 26
    • View Profile
Working on Unicode Support
« Reply #8 on: February 18, 2006, 05:19:47 PM »

Unicode is a good concept but when you actually try to implement it in code it becomes quite complicated.  As far as I could find, this code that has been implemented in OIS has never been posted on the net.  I googled both web pages and usenet groups and by putting various bits of information together came up with this solution, which works so far.

To implement Unicode on other platforms (Unix or Mac) I could see the need to implement the FoldString function such that a lookup table converts two keystrokes into one, such as looking up the acute accent and seeing to which letters it can combine: é, É.  This would mean that the Unicode table (the link provided with the code) could be used to derive all possible combinations, much like I did to support the french letters in the current implementation.
Logged

cdsnyder

  • Regular
  • *
  • Karma: +0/-0
  • Posts: 26
    • View Profile
Working on Unicode Support
« Reply #9 on: February 19, 2006, 09:49:17 AM »

I'm kind of lucky in that Mac has a lot of international support built *into* the system instead of just on top of it. I've got unicode support  for almost all the languages I've tried. I can change my keyboard layout on a per program basis, and it also gives me an overlay window with the new language for reference, so testing is easy. I've tried french, german, and some japanese palattes. I could not actually overlay languages that use specialized keyboards, unfortunately, so I had to use a popup palatte for Asian alphabets. Dead keys for accents are working fine. OSX simply sends a keydown event for the deadkey press, but with no unicode string, just the keycode. When the key ending a unicode sequence is pressed, it sends that string in unicode with the keydown event that terminated it, and I get accented characters. The only one's I've been unable to get working are the chinese/japanese setups that translate a phonetic spelling into a pictogram.

Now that I'm a little better versed in this and have done some testing... I would really suggest making an array out of the unicode text passed through an event. I was outputting the unicode returned by events, along with the length of the unicode string. Mainly it was the East Asia palettes, but on occasion I did recieve lengths of 4 for a single character, which is 3 wide chars + terminator. It is still rare, but it does happen.  IMHO things that prevent full unicode support in the future are kinda pointless, and changing it now would be much easier than changing it after it has been established for a while. Maybe just setting it to text[UNI_MAX]  and a #define or const int would make it easy to change in the future if those rare cases pop up. Overhead will still be very low, compared to a wstring or the like.
Logged

pjcast

  • Administrator
  • Veteran
  • *****
  • Karma: +1/-0
  • Posts: 2661
    • View Profile
    • http://www.wreckedgames.com
Working on Unicode Support
« Reply #10 on: February 19, 2006, 11:16:18 AM »

I think, that this would be best handled by repeating the keyDown event multiple times for each unicode value. If you get 3 or 4 unicode values, just call that three times. The app could handle repeated keydowns for something like this. I can just make a note that some KeyCode generates multiple keyDown events in certain languages when TextTranslation mode is set to Unicode (as opposed to Off or Ascii).

The Mac sounds like a sweet platform to be giving you all that info.. Trying to pull that information out of Win32 API is liking biting on nails. :axe:
Logged

cdsnyder

  • Regular
  • *
  • Karma: +0/-0
  • Posts: 26
    • View Profile
Working on Unicode Support
« Reply #11 on: February 19, 2006, 11:56:48 AM »

Ok, should we go ahead and reserve some keycode as a null keycode, like 0x0? So that it initiates the first unicode char with the correct keycode, and additional ones are null. Otherwise the multiple keydowns may be incorrectly interpreted by the program as control movements. As if it will ever happen hehe... but just to be thorough.

I'm not surprised windows is a pain.. that's why I don't have any windows machines in the house anymore hehe. Unfortunately, it is probably 90% of what people are using. I'm sure I'll end up dealing with it again sooner or later.

I haven't looked at linux's international support at all yet. I would hope they made things easier, since so many countries contribute to it.
Logged

pjcast

  • Administrator
  • Veteran
  • *****
  • Karma: +1/-0
  • Posts: 2661
    • View Profile
    • http://www.wreckedgames.com
Working on Unicode Support
« Reply #12 on: February 19, 2006, 12:12:45 PM »

Finding the right way to do things with X11 is not easy for a linux novice like me. But, there are methods I'm sure. However, I generally like the Win32 API over the Linux ones :D (especially because msdn has lots of info.. man pages are sometimes lacking). The bigger problem is that there are no decent examples to be easily found for Win32 or Linux to handle this effectively.

I don't think multiple key down events could hurt an app when they are in Unicode mode (when in Off or Ascii TranslationMode, I don't see how they could ever recieve more then one character with a button press). If they are recieveing text, it may be useful to know which key_code was recieved at the same as knowing what unicode value is being recieved.

And, when inserting keyDown events into CEGUI (For example) only control keys are acted upon (Up, Down, Enter, etc) and these keys would not be repeated anyways, as they should not be generating any unicode (or ascii) characters. And, a user should not be really in text mode when in regular gameplay (and not processing text). And, even if they are processing text, and they are also acting on key down events for game motions, a second key down event should not cause any problems. Usually you set a flag to true in the key down, and only set it to false in the keyup. So, I propose not adding a null key for this rare special case. It would just mean that the user would always have to check for that null which would not generally make any difference imo.

as a side note:
Some things that caused problems with this before were auto generated KeyPresses in X11 - key repeat defaults to on for the system, and there is no consistant way to know if a key is repeated. So you end up with keyDown, keyUp, keyDown, etc etc when holding down a key. Does OSX send repeated key messages when holding keys down? And if so, did you take that into account? It was a real problem on Linux until I disabled key repeat (with an X Function call, that also sets the whole system to off - so, if the app crashes and does not reset it, the system no loner does key repeats :D ).
Logged

cdsnyder

  • Regular
  • *
  • Karma: +0/-0
  • Posts: 26
    • View Profile
Working on Unicode Support
« Reply #13 on: February 19, 2006, 12:37:52 PM »

Oh ok. I was in the wrong frame of mind about the use of the TranslationMode. I was thinking it was kind of a one-time set. Then whether recieving events for movement or text, the person would just check the key or text. If I understand correctly now, it was meant to act more like a state machine, right? When in control mode, set it to Off. When expecting text, set it to Ascii or Unicode.

I handled repeat events this morning, I haven't commited the changes yet. I made it an option in the paramList like you did with X. OSX had separate events for repeat keys. So if it is enabled, I register an additional listener for repeatkey events and send additional keyPressed calls for each one.  The OSX keydown and keyup events will always correspond to the physical pressing and releasing of a key, regardless of any repeatkey events.

I meant to ask about the handling of repeat events earlier, but forgot. I set it such that  only keyPressed gets called multiple times for repeat keys. Should a repeat event send multiple keyPressed calls or multiple keyPressed + keyReleased?
Logged

pjcast

  • Administrator
  • Veteran
  • *****
  • Karma: +1/-0
  • Posts: 2661
    • View Profile
    • http://www.wreckedgames.com
Working on Unicode Support
« Reply #14 on: February 19, 2006, 12:51:57 PM »

Yeah, Off, Unicode,and Ascii is like a state machine and can be changed at anytime reducing the overhead of translating the character.

Multiple keyDown messages is the best way to handle repeats if enabled. Because, this is also the only place that the tranlation should be done (on Windows and Linux it is a little extra oevrload to tranlates the charcter, so it is only done once).

A thing to note about key events, they were not originally part of OIS. DirectInput (used on Windows) does not send repeated key events for held done keys. But, on Linux using X11, you will continue getting X Events when holding down a key. Though, you get a keyDown/keyRelease pair multiple times, which makes detecting it as a repeated key difficult, it could be the user hitting the key really fast. So, by default, it is disabled, the extra param is just in case the user did want to made there code a little OS specific behavior for X Repeats.

I think, in the next release of OIS, that there will be an added setAutoRepeat( int someTime = 0 for off ) where the user can turn off, or set a time value for when repeated keys will be sent. Which would result in multiple keyDown events, not a keyDown and then keyup (since the key would never be up). This would be useful of course for text enry where the user wants to hold down the key (usually the arrow keys are used mostly for repeated keys). However, on Linux, I will not use the standard X events, as they would result in keyUp messages, and DirectInput does not send repeated keys. So, on those platforms it would be a library internal method that would be done in the Capture method if enabled. Though, it would also mean more code and the use of system timers. So, i think a 0.6.0 release is better for that.
Logged
Pages: [1] 2