SQL Server will use up to the max memory defined for that instance - if that instance uses the default then SQL Server could use up all memory on the server.
If you have multiple instances on the same server - you have to set a max memory for both instances that total less than the available memory on the server, leaving enough memory for the OS to do its work.
With 12GB available on the server you should not exceed 10GB of memory combined. If your systems utilize CLR, SSIS, SSRS or any other applications then you need to account for those systems memory requirements.
With multiple instances it is also a good idea to set a minimum memory value - which is the minimum amount of memory that instance needs in order to perform adequately. If this is not set then the other instance could force that instance to give up more memory than it requires to function correctly.
And finally - remove any settings related to lock pages in memory if you are going to allow the multiple instances to steal memory from each other. For example, if you set the first instance to 6GB and the second instance to 10GB max memory - allowing the second instance to use up all available memory if needed, having locked pages in memory will block the second instance from gaining that memory once the first instance has allocated it...