While there are folks who can answer the displacement or stability issue, and the problems with windage, I may be able to at least suggest some ideas at least using light building materials.
In the past, along these lines, I've wondered about using some of the skin on frame boatbuilding techniques (over a light aluminum frame with ribbing of wood good SOF building) for the walls of such a house, both for the outer and inner shell. providing them insulation and including an air gap mid insulation for soundproofing (maybe low density closed cell foam would work).
Windows should not be of ordinary window glass but of thin glass glued to a plexiglass sheet or just plexiglass.
The roof is the main place you may need to worry about things like hail and here a variation of something called Gilwood may be of interest: Gilwood is a form of plywood where wire mesh is used between the layers of wood and is very tough and puncture resistant. It would be easier to, along these lines, use very thin sheets of plywood for the layers and a modern material like Kevlar for the reinforcement. This seems like it may be important: you will need to have your reinforcement nearer the surface than near the middle, so 3 layers of thin ply and two of Kevlar (or whatever you choose). The top layer needs to be the hard/tough layer, while I would imagine the other layers mainly need to resist delamination and deformation, so maybe a good foam would work for them? Note: this probably wouldn't be a roof to walk on (me = Captain Obvious).
Building light certainly seems a way to mitigate any issues that may arise.
