It appears that ScrollView cannot be scrolled using direction keys, so I tried binding Actions to directional keys and do the scrolling manually.
I thought this would be possible by altering viewport.x and viewport.y, but nope :)
Any ideas how to do it?