The coefficient of thermal expansion for stainless steel differs depending on grade, as does that for aluminium depending on alloy, however, for comparison, most grades of stainless steel will have a higher coefficient than most aluminium alloys.
In the real world,on a cold day you'd just choose a larger insert, as the day warmed up you'd switch to a smaller one, assuming the sizing of the paint remained constant, which it wouldn't, For best practice you'd resize as ambient temperature changes.
Brass, would typically have a coefficient of thermal expansion somewhere between the two, dependant on the zinc to copper ratio of the material. The downside of brass as I see it is you'd need to keep a couple of spare fluffies and a pot of metal polish to keep them clean as brass oxidises quite quickly (Obviously stainless steel doesn't and aluminium inserts are anodised to prevent corrosion) This would, over an extended time period affect the sizing fractionally.
For the ultimate insert material use titanium, it has a lower coefficient of thermal expansion than stainless steel, naturally forms a protective oxide layer preventing further oxidisation (or can be anodised) has a better strength to weight ration and, as an added bonus, you may well get to see sparks fly from your barrel in low light conditions! However, they would be more difficult to manufacture and more expensive.
As far as weight goes, titanium inserts would come out at about half the weight of stainless steel ones, whereas aluminium inserts would come out at about half the weight of titanium.