Is it OK to set AVcc to GND on the H8/3297 ?
No, this cannot be done. The analog circuitry is supplied from the AVCC power pin. It is impossible to use this pin as a general input pin. The A/D Converter has internal level shifting circuitry used for adjusting control signal levels between the analog and digital circuitry. If Avcc=0V, this circuitry may operate incorrectly.