It depends on several variables, the most important one: what type of water heater you’re talking about… I’m assuming it’s a water heater since you posted it in WH Venting category.
Natural draft – very hot initially, right after the burner ignites. If you have a proper draft it should cool down considerably in a minute or so but still hot. If there is a double wall / B-vent starting at the draft hood, its outer wall shouldn’t be hot at all.
Power vented (PVC pipe) – warm
Tankless – depends on exhaust system and material used