Similar experience with matching ATF36077 preamp with older WA8NLC transverter (originally built in 1991!). Recently recycled this transverter from the junque box for VK5AKM's home station.
This one still had the original two MAR6 & MAR1, these were prone to oscillation at lower frequencies with all their gain peaking below 1 Ghz. I removed the old MMIC's ahead of the filter and replaced with a single ERA3. initially had some stability issues with the ERA 3 so improving the grounding around the ERA3 fixed that. (this is the original board that didn't have plated through holes!). Adding the pre amp you could still provoke it to oscillate with your finger on the input of the ERA3. I initially modded the ATF36077 pre amp with a small smd resistor in the drain (think its 10 ohms) but eventually added a 3db pad between the preamp and transverter. That fixed all.
While doing mods I though I'd sweep the RX hairpin loop filter. As expected found it was dimensioned for 3456 MHz but with the centre frequency around 3480 Mhz. That meant at 3400 Mhz the through loss was 8 db and return loss was poor. I decided to have a go at shifting the bandpass using some different dielectric materials on the filter. I settled on the right thickness of celluoid to drop the passband to 3380 - 3460 MHz. Superglued the patch on (the glue didn't shift the response!) R/L was much better and image rejection ~ 40 db (plenty).
Did the same experiment with the Tx filters. In the original version one filter is fine (centred around 3420 Mhz) the other was the same dimensions as the Rx filter so gave that the same celluoid treatment. I don't have experience with the new DEMI transverters, suspect 3400 Mhz would have been a consideration in newer designs and may be ok however worth investigating
The end result is snap .. another Purple Stealth based 3400 Mhz transverter!
Inside shot the old WA8NLC transverter is mounted under the PCB sub chassis. The 1085 Mhz PLL with OXCO is in the small diecast box, EME66 switching and ATF36077 preamp on the top. An old CB transceiver meter provides power out indication
Cheers
David VK5KK