There might be a clever way to use the order by clause in the LAG function to get the result you want, but the light bulb is not going on in my mind. You can always use a subquery like shown below which will always work, but is not as clever and not as efficient as using the LAG function.
SELECT a.*
,b.LoadingDate
FROM
My_Table a
OUTER APPLY
(
SELECT TOP (1)
b.LoadingDate
FROM
My_Table b
WHERE
a.ProductKey = b.ProductKey
AND a.GrandCustomerKey = b.GrandCustomerKey
AND ( a.column_a > 0 OR a.column_b > 0 )
AND a.Date < b.Date
ORDER BY
b.Date DESC
) b;