From what I can tell, it used either special hardware or math.
There used to be this great game on the Mac, Spectre VR, which had filled polygon graphics, but required no 3D accellerator card. (It's sort of like Battlezone.)
http://en.wikipedia.org/wiki/Spectre_%28video_game%29
I'd make a Gameboy port of that.
Or maybe a GameBoy PDA cartridge with a keyboard-cradle thing. (Think the keyboard on a PS2 controller, but on the bottom of the GameBoy.) Of course you could print out notes and stuff with the Game Boy Printer.
Maybe a music player for Game Boy? It could pull all the music off an SD card and pipe it through the speakers / headphones. (I'd imagine the cartridge would have to do most of the decoding.)
How about a music composition cart? It could connect through USB to the PC so you can dump MIDIs on it. Maybe it could connect to another gameboy with the same cartridge to get a Gameboy duet going...
Coming up with silly ideas is fun!
Maybe I'll put together some screenshots when I feel like drawing.
