PORTS: IOS: Apple keyboard cursor keys don't work, accessibility bar covers bottom of screen

[Purduecoz]

This issue seems to be similar to previous keyboard issues with the iOS port.

I built ScummVM stable (2.3.0pre) and master and installed them onto the GM/RC of iPadOS/iOS 15. Using the Apple Magic keyboard causes an accessibility bar to cover the bottom inch or so of the screen, which affects most games that don't have a parser/input bar at the bottom (it is only just barely below the parser line for the AGI engine, and with the SCI engine, it covers the bottom of the screen). There is a minimize option accessible from the bar that minimizes it to a button in the lower right corner which helps greatly.

The bigger problem is that the left/right/up/down arrow keys do not work, nor do Control modified keys (like Control-Q to quit in the SCI engine). Letters, numbers, and return seem to work fine.

[criezy]

I only have a quite old iPad that is stuck to iOS 12, but with that one I checked that the arrow key from an Apple bluetooth keyboard work as expected in King's Quest II. And I have no idea why this would not work on iOS 15 or with other keyboards.

One thing to keep in mind is that the keyboard needs to be enabled in ScummVM (for example with a pinch in gesture) before it can be used. But since you mention issues with the accessibility bar at the bottom of the screen, I assume you knew that already.

While testing the arrow keys I noticed that the escape key from the keyboard was not working. And it was indeed never implemented; we only add support for using the ` key as escape (which is useful for keyboards that do not have an Escape key). But now if there is an Escape key it also works properly.

Shortcut with modifiers was never implemented, so it is expected that they do not work. Implementing that one a case by case basis would be possible, although not trivial, but we cannot do it for every possible keyboard shortcuts. I am not sure if there is a way with the iOS API to handle generic key press with modifier.

Regarding the "accessibility bar", I cannot reproduce either, but then I am not sure what you really mean. I have never heard of an accessibility bar, so I don't know if this is something new in recent iOS versions, or if you mean either the input assistant bar or input accessory bar.

• The input assistant bar is a standard bar that typically has undo, copy and paste buttons.
• The input accessory bar is a custom bar that applications can implement. In ScummVM we implemented such a bar to have function keys, arrow keys, escape keys and more.

I have attached screenshots for those two. Can you clarify if you mean one of those two or something else?

In ScummVM we try to always hide the input assistant bar. This only works on iOS 9 or above (which is your case) when compiled with the iOS 9.0 or above SDK. I assume that this is also your case. You can see ​https://github.com/scummvm/scummvm/commit/3baad9e2a for details. If that is the bar you see, we may have an issue with this code. The UITextInputAssistantItem documentation suggests that setting the leadingBarButtonGroups and trailingBarButtonGroups to nil as well as setting the autocorrection type to UITextAutocorrectionTypeNo should hide the bar. And I just noticed that while we do the later, we set leadingBarButtonGroups and trailingBarButtonGroups to an empty array and not to nil. That works on iOS 12 at least, but maybe that matters for iOS 15. Or maybe there is something more that needs to be done. But not having such a device this is not something I can experiment with myself.

We also try to hide the input accessory bar if a physical keyboard is connected. They is no API to know if a physical keyboard is connected so, so we have to use some guesswork. You can see some details on ​https://github.com/scummvm/scummvm/commit/cfd66173. The code that matters is at the bottom in prepareKeyboard. Basically it looks at the height of the keyboard frame. Because when a physical keyboard is connected only the bars are visible, the height of the frame is a lot smaller. So we assume that if this is less than 140 point we can hide the bar. If this is the bar you see, maybe this guesswork does not work on iOS 15.

But I am actually wondering if it might not be better to have a user option to show or not the input accessory bar instead of some guesswork to show it when there is no physical keyboard and hide it when there is one. My understanding is that iPad keyboard does not have function and escape key. So users might want to see this bar even if they have a physical keyboard (and it also has a key to show the ScummVM menu).

[Purduecoz]

To answer your initial question, yes I am aware the keyboard does need to be activated. When I do activate it, the bar I was describing pops up, which is neither of the bars you’re describing, but is always present when I am inputting text, as shown in the attachments I’ve added. It seems to be a dictation or autocomplete shortcut bar that is trying to guess words that I am typing.

I did notice that hitting the arrow keys seems to cycle through previous letters and words that I’ve entered into the parser, but does so on this autocomplete bar - not in ScummVM. I’ve tried to add another attachment that shows this in Space Quest 2. Hitting the arrow keys appears to be intercepted by iOS and cycles through these letters/words in the ‘autocomplete bar’ instead of moving the character on screen.

My Apple keyboard does not have escape, but the ` button functions correctly as such.

Certainly willing to test any ideas or potential changes in order to help troubleshoot. It seems like the arrow keys are being intercepted by iOS as long as this input bar is present on screen (which I cannot figure out how or if I can turn off).

[Purduecoz]

I figured out that it seems to be called the predictive keyboard, and turning it off in the Settings, keyboard menu in iOS does make it disappear, but doesn’t restore movement with the cursor keys. The return and ` keys continue to work as expected though.

[Purduecoz]

One other thing, I did use a keyboard checker website in Safari on my iPad to verify that the arrow keys were registering as the correct arrow keys. Just trying to eliminate possibilities.

[Purduecoz]

Just had a chance to test a couple other Bluetooth keyboards on the iPad; they all exhibit this problem when the iPad is running iPadOS 15. They all worked properly on iPadOS 14.x.

[criezy]

Thank you for the additional information.
So the bar you see is actually the input assistant bar. It contains by default copy/paste buttons, auto correction and predictive text. In ScummVM we try to remove all of those so that the bar is not visible.

Setting the autocorrection to UITextAutocorrectionTypeNo used to disable both the autocorrection and predictive text. But apparently that is no longer the case in iOS 15. I don't know if the change was done on purpose or if this is a bug in iOS 15, and I can't find anything new in the Apple developer documentation that could be used to disable the predictive text from an application. But at least, as you found, there is a workaround as the user can disable predictive text in the iOS keyboard settings.

I suspect the arrow keys issue is related to the new wantsPriorityOverSystemBehavior property for a UIKeyCommand that was added in iOS 15. To test this theory can you please edit backends/platform/ios7/ios7_keyboard.mm to have the following code and see if that fixes the issue for you?
The code starts around line 150.

backends/platform/ios7/ios7_keyboard.mm

@@ -148 +148 @@
 
 - (NSArray *)keyCommands {
        UIKeyCommand *upArrow = [UIKeyCommand keyCommandWithInput: UIKeyInputUpArrow modifierFlags: 0 action: @selector(upArrow:)];
+       upArow.wantsPriorityOverSystemBehavior = YES;
        UIKeyCommand *downArrow = [UIKeyCommand keyCommandWithInput: UIKeyInputDownArrow modifierFlags: 0 action: @selector(downArrow:)];
+       downArrow.wantsPriorityOverSystemBehavior = YES;
        UIKeyCommand *leftArrow = [UIKeyCommand keyCommandWithInput: UIKeyInputLeftArrow modifierFlags: 0 action: @selector(leftArrow:)];
+       leftArrow.wantsPriorityOverSystemBehavior = YES;
        UIKeyCommand *rightArrow = [UIKeyCommand keyCommandWithInput: UIKeyInputRightArrow modifierFlags: 0 action: @selector(rightArrow:)];
+       rightArrow.wantsPriorityOverSystemBehavior = YES;
        UIKeyCommand *escapeKey = [UIKeyCommand keyCommandWithInput: UIKeyInputEscape modifierFlags: 0 action: @selector(escapeKey:)];
        return [[NSArray alloc] initWithObjects: upArrow, downArrow, leftArrow, rightArrow, escapeKey, nil];
 }

Mapping the ` key to Escape has indeed been in place for a long time so that when using either the virtual keyboard or a physical keyboard without the Escape key it was possible to type Escape. I still think it could be a good idea to add an option so that the user can enable our custom input accessory bar even when using a physical keyboard. Many DOS games used functions keys for example for loading or saving, so having access to those can be required for some games. So I might look into that as well.

[Purduecoz]

Your hypothesis was spot on; making the changes that you provided above resolved the issue. Thanks so much for taking a look into this!

[criezy]

I have now pushed a change that should fix the arrow key issue.
The fix had to be a bit different from what I asked you to try so that we can still compile ScummVM with older SDKs and run it on older iOS versions.

Can you please test it and confirm that it works now?

[Purduecoz]

I did not see the ticket update and haven't installed the latest version onto my iPad. I will do that and report back. Apologies for not being more timely!

[Purduecoz]

I can confirm that the fix is working great with ScummVM build 2.5.0 on iOS 15.2. Thanks again!

[bluegr]

Great, thanks for the update!

Closing