very high BER (~13%)

[DG0OFZ]

Hi,

Tried changing the RX offset from 1000 down to -1000,
Tried changing the RX level from 5% to 100%,

With the design of the repeaterbuilder boards you can't change RX or TX frequency by the modem.
The modem is pure audio processing.

Please check your external radio for correct Rx and TX frequencies!
Also check the RX bandwidth of the external radio.

[DG0OFZ]

Hi,

The HT TX is at 146.0391MHz (had to use in analog mode to keep it active long enough to get a reading), the RX is programmed for 146.040 MHz/156.7.

So you are 900Hz off. I'd think that's to much.

Please make sure, that your TX and RX are on the correct frequencies. This means measure the frequencies!

With (pistar-)mmdvmcal you can set a constant carrier without modulation. So you are able to connect the TX to a counter (please: not directly!) and see if it is TXing on the right qrg.
For RX you can't measure it so easy. But you may check for lowest BER while your repeater is in RX mode an you are TXing with another device.

[AF6VN]

The HT TX is at 146.0391MHz (had to use in analog mode to keep it active long enough to get a reading), the RX is programmed for 146.040 MHz/156.7. I am not able to adjust the settings in the RX radio at the moment, waiting for the programming box to get delivered later this week. I have picked off the raw audio inside the radio so I don't think the PL will be an issue for my purposes.

For a dual radio (repeater) configuration, those frequencies look rather close together -- such that the Tx could be feeding back on the Rx radio.

I have dug out my Agilent 53131A (225MHz capable) and used an antenna for the input. When I key the HT in analog mode, I get a reading of 146.040.0xx.

Using a similar technique, I am measuring the TX frequency on the receiver (assuming the same oscillator effects will be in play at RF frequencies for Rx and Tx) and I am getting 146.640.1xx.

... while that appears to reflect common 2m repeater offset /between/ Tx and Rx phases. I'd think one of the radios should be listening on 146.6xx while the other is transmitting on 146.0xx -- as the system user would be sending on 146.6xx and listening on 146.0xx (or the opposite configuration depending on + or - repeater shift setting).