The thing is, language is a combination of details: words, intonation (or pitch in some cases), accent, gestures, speed, eye-contact, etc.
Written language is neccesary to preserve, therefore there are some rules about how we’re supposed to write. Orthography and grammar are both historical results, they’re traditions like church on Sundays or trying to kiss the girl you like for the first time at the end of your date when you drop her at her flat.
Grammar of course is applied to oral language as well, more as a habit (like the aforementioned examples), but then of course vocabulary is more important. It’s the same old story many language teachers tell, but this actually happened to me when I was working in Tokyo: once a person asked me, in horrible Jap accent, something along the lines of (and I’m writting that phonetically) “suté-eshon tulen fua?” (station train where?). But I understood him and I could tell him in a not-less-hideous Japanese how to get there (I suppose I used similarly ridiculous pronunciation but he understood me anyway).
Some other time, a wakai-otokono-hito asked me, in a beautiful and meticulous British accent “can you please tell me how can I get to the … to the … to the…”. His grammar was perfect, his pronunciation was worth recording (he even spoke better than many Londoners I’ve met), but he couldn’t tell me where on earth was he going. We were on a packed area so he could be looking for anything (a store, a cashpoint, a park…). I asked him to tell me in Japanese and I couldn’t understand him.
That’s why I think vocabulary comes first. Still grammar affects certain situations (“what do people think” has got different meaning to “what people do think”), but its use is more out of tradition than anything else imo.
As for your experiment, it’s not that easy: if your friends had been born and taught how to communicate with so-called gibberish, they could understand each other that way because, for them, each sound would have a different meaning. That’s how crows speak to each other. For us it’s nonsense, for them it works.
If it’s working so far with your mates, it must be mere accident, or as somebody suggested, cultural correspondence.