That's what happened to me too, I changed the rear caliper along with the pads.
I found a video, anyone tried this?
https://www.youtube.com/watch?v=DcAwV0wd92A
Fast forward to 3:40. In the comments he mentioned his pad chatters and this homemade shim solved his problem!