I find the difference to be negligeble, especially when mixing 1/32 aircraft or helos with 1/35 tanks and APCs, or with civilian vehicles that are 1/32 with 1/35 armor and figures. Like Jaster says, as long as you keep constant sized items the same, weapons, ruck sacks, canteens, etc. you will be fine. As long as it looks good to you, go with it. Here are some examples of 1/32 cars mixed with 1/35 armor and figures.
1/32 Tamiya Mini-Racer Land Cruiser body on scratched up frame, running gear and interior:
1/32 VW pick-up, Tanker truck, and Chevy Impala that started life as a police car:
1/32 VB Bug w/ 1/35 tank and figures:
1/32 F4 Phantom with 1/35 M151A2, also one of the pilots is a 1/32 figure, the other is 1/35: