Because CLAIMS.[TOTALPAID] is numeric your ELSE clause needs to be numeric too - you have a blank string which is what is raising the error. Note the message: "Error converting data type varchar to numeric". Change it to:
,max(case when patins.rank = 1.0 then CLAIMS.[TOTALPAID] else 0.0 end)
and that should fix it.
In the second case INSCOMP.ORG is datatype String (Varchar / Char / similar) so it is OK to have a Blank String in the ELSE.
Sometimes SQL will make an implicit conversion - for example Blank String to INTEGER is usually OK, but it will not do that to a Numeric datatype.
Maybe the ELSE was intended to prevent MAX giving a warning if it encounters NULL values? (We deliberately avoid that situation here)
Might also be a problem with the APP displaying the values if it is asked to display NULL - e.g. maybe the user would prefer to see "zero" rather than whatever the APP displays for NULL - which might be "blank". A solution for that would be to use a COALESCE() or NULLIF() to force the MAX result to 0.0 when it is NULL