i see what you mean as being kinda not good at anything
No, not necessarily. All design involves tradeoffs. With a small, light vehicle you have more wiggle room than with a larger, heavier one.
If you want to race and be competitive in autocross or off-road events, the tradeoff would likely make one mode almost useless. But if you just want to go out and have fun, I'm not seeing an insurmountable problem.
One of my first motorcycles as a teenager was a Kawasaki 175 Enduro. A '73, back when the front and rear tires were the same size. I put 16,000 miles on it in 14 months. After that the "enduros" became increasingly specialized dirt bikes, less and less streetable. Mine was tall and gawky for a street bike, too low for some kinds of off-road, had the extra weight of the street gear, halfbreed not-quite-knobby tires, etc. But I could (and often did) veer off the road, across the ditch, and down some deer trail, and I had an absolute blast with that thing.
People always tried to describe the bike by what it wasn't - it wasn't a real dirt bike, it wasn't a real street bike, it wasn't lightweight, it wasn't powerful... I preferred to look at is was what it was - it was able to make 200-mile freeway jaunts, it was able to climb any hill the other guys could, it was able to get 60mpg, it was able to do anything I asked of it. Well, except for the time I guessed wrong about how deep a ford was, and it wound up going home in the back of a pickup and we took the motor apart to get all the water out.
Back to your idea...you might want to take a look at available tires. They make slicks and street tires down to 22" tall, and I'm not sure how large off-road tires get nowadays. Pick a ride height - say 5 or 6 inches for the street, with 22 or 24 inch tires. Look up available off-road tires that aren't too wide, and see how much change you get in ride height. Enough? Juggle your options and roll the dice again.
Remember, "perfect is the enemy of good enough."
And as far as being competitive... you've probably heard it before, but attend to the parable of the two swimmers who were chased by the bear.
One took off running, the other stopped, rooted around in his pile of clothes, and started pulling on his shoes.
"You idiot, you can't outrun that bear!" says the first one.
"No, all I have to do is outrun you
!" says the other, tying his laces...