As far as I know dosbox uses XMS 2.0 which is limited to 64MB
see :
https://en.wikipedia.org/wiki/Extended_memory#eXtended_Memory_Specification_.28XMS.29
To get up, we need probably update sources to 3.0, but there is no official patch 🙁 For a interest I tried expand ram support to 256MB, however while expanded memory was raised correctly, extended still fails. I can look at this yet, I must find free time, but this feature will not be included to official svn or normal core.