I haven't played with that system recently but AFAIR, you should be able to build the sysroot for your system on your desktop machine. Then you could use scp to copy the bits you need over to your device.
Doesn't your system provide a possibility to start from another device like a SD card or even a nfs ?