Invalid column name

Andy,
There are many situations where I have used SET QUOTED_IDENTIFIER OFF safely. As you point out, there are many situations where it can't be used. Awareness of which features require the ON setting are few enough to become familiar with. As with anything you do, ignorance can bite you if you don't study the effects of the settings you use. It isn't a legacy setting. It is supported in SQL Server 2017. You may be over stating the warning but it's good to become acquainted with it's usage and side effects.
Don't SET QUOTED_IDENTIFIER OFF if you're:

  • Executing DDL on an index

  • Creating an indexed view or adding a computed column

    • These are the required settings for indexed views and computed columns: SET ANSI_NULLS, ANSI_PADDING, ANSI_WARNINGS, CONCAT_NULL_YIELDS_NULL, QUOTED_IDENTIFIER ON; SET NUMERIC_ROUNDABORT OFF
  • Invoking XML data type methods