Re: 64-bit problem with VM statistics
Re: 64-bit problem with VM statistics
- Subject: Re: 64-bit problem with VM statistics
- From: Tony Scaminaci <email@hidden>
- Date: Wed, 1 Jun 2005 13:56:08 -0700 (PDT)
This is the "memtest" utility - I'm the author for OS
X. It runs in single-user mode as well as mutlituser
mode and tests nearly all the installed ram in the
system (within about 97.2% of the currently free
amount as reported by vm_stat). Most people run it in
single-user mode so they can test virtually all of the
installed memory. I know some people at Apple are
aware of this app already because the algorithms have
found memory and cache problems where Apple's hardware
tests have not.
Under Panther, I couldn't malloc more than about 2 GB
of ram at one time. I chalked it up to some limitation
in malloc lib and did some heroics to prevent machines
from generating tons of malloc errors if they had more
than 2GB of installed ram. Going further and
attempting to mlock nearly all the free ram would
cause the kernel to become unresponsive, requiring a
hard reset. Over a long period of time and with a
bunch of testers, I found that if I limited the
allocation to 97.2% of the reported free memory in
single-user mode, the kernel would not lock up. I
don't know why this was the case but it simply was.
It's a well-known issue under both Panther and Jaguar
and I've been battling it for over a year. I even
filed a bug report in radar last year.
I've been getting increased requests to support up to
the full installed ram. Since Tiger's release, these
requests have gone out the roof because more people
are installing 4-8 GB in their Macs. They've been
complaining (loudly) that they couldn't test more than
2G of ram in one run under Panther but that limitation
magically disappeared under Tiger. My beta testers
found they could grab up to the 4G limit in both
single-user and multiuser modes but couldn't go beyond
that because the GetFreeMem() function was only
32-bit. So I changed it to be 64-bit clean or so I
thought. I've made all the changes I'm aware of to
fully support 64-bit mode but this one problem
remains. I've got over a dozen reports of G5's running
Tiger completely locking up if they try to test all of
their memory. I assume that malloc is trying to
allocate terrabytes of VM since the Avaialable Free
Memory number is so outrageously large. Whatever the
issue is, it's in this function and that function
determines how the rest of the app works or doesn't.
Memtest is available at
http://www.memtestosx.org/download if you want the
entire XCode2 project and it's licensed under GPL2. I
need to understand why the typecasting isn't working
correctly in the GetFreeMem function and fix it
quickly. I don't have a 64-bit platform to test on so
I'm relying on user feedback and testing. Whatever
help I can get is greatly appreciated.
Thanks,
Tony
--- Kevin Van Vechten <email@hidden> wrote:
> Thanks, but I was hoping for a reference to the
> project so we can all
> be on the same page. Are you looking in xnu
> sources? In
> dynamic_pager sources in system_cmds? Which file?
>
> Could you elaborate how a large number can cause the
> system to lock
> up completely? i.e. what's the actual operation in
> question?
>
> - Kevin
>
> On Jun 1, 2005, at 1:03 PM, Tony Scaminaci wrote:
>
> > Kevin,
> >
> > I've gotten feedback from a number of users that
> the
> > large number being reported also causes their
> systems
> > to lock up completely. It's not just a printf
> problem.
> >
> > Here's the source code for the function in
> question:
> >
> > unsigned long long GetFreeMem()
> > {
> > vm_statistics_data_t vm_stat;
> > mach_port_t host_priv_port,
> host_port;
> > mach_msg_type_number_t host_count;
> > kern_return_t kern_error;
> > unsigned long long FreeMem; //
> Free real
> > (physical) memory
> >
> > // Get host machine information
> >
> > mach_port_t get_host_priv()
> > {
> > return(mach_host_self());
> > }
> >
> > mach_port_t get_host_port()
> > {
> > return(mach_host_self());
> > }
> >
> > // Get total system-wide memory usage structure
> >
> > host_priv_port = get_host_priv();
> > host_port = get_host_port();
> > host_count = sizeof(vm_stat)/sizeof(integer_t);
> > kern_error = host_statistics(host_priv_port,
> > HOST_VM_INFO, (host_info_t)&vm_stat, &host_count);
> > if (kern_error != KERN_SUCCESS)
> > {
> > mach_error("host_info", kern_error);
> > exit(EXIT_FAILURE);
> > }
> > host_page_size(host_port, &mac_pagesize);
> // Get
> > page size for this machine
> > FreeMem = ((unsigned long long)
> vm_stat.free_count)
> > * mac_pagesize; // Calculate total free memory
> in
> > bytes
> > FreeMem &= 0xFFFFFFFFFFFFFFC0ull;
> // Align byte
> > count to 64-byte boundary
> > return(FreeMem);
> > }
> >
> > I need to have the correct number of bytes
> returned to
> > the calling function in both 32-bit and 64-bit
> modes.
> > This code works correctly in 32-bit environments
> but
> > is breaking on 64-bit platforms.
> >
> > Thanks,
> >
> > Tony
> >
> > --- Kevin Van Vechten <email@hidden>
> wrote:
> >
> >> Perhaps the problem is with the format string
> used
> >> to print the
> >> number instead of with the number itself?
> >>
> >> Would someone mind pointing me to the specific
> >> source file in question?
> >>
> >> Thanks,
> >>
> >> - Kevin
> >>
> >> On Jun 1, 2005, at 6:23 AM, Tony Scaminaci wrote:
> >>
> >>
> >>
> >>>> From: Tony Scaminaci
> <email@hidden>
> >>>> Date: June 1, 2005 8:18:31 AM CDT
> >>>> To: William Kucharski <email@hidden>
> >>>> Cc: Tony Scaminaci <email@hidden>
> >>>> Subject: Re: 64-bit problem with VM statistics
> >>>>
> >>>> Unfortunately, this is the exact line that's
> NOT
> >>>>
> >> working on 64-bit
> >>
> >>>> platforms. FreeMem is returning with the
> >>>>
> >> following:
> >>
> >>>>
> >>>> Available memory: 268267683840MB
> >>>>
> >> (281299054850211840 bytes)
> >>
> >>>>
> >>>> The number of bytes (which is what FreeMem is
> >>>>
> >> holding) is way too
> >>
> >>>> large. It should be around
> >>>> 4613734400 bytes on this particular machine
> which
> >>>>
> >> has about 4400
> >>
> >>>> MB free as shown by top.
> >>>>
> >>>> I received a bug post from John Caldwell who
> >>>>
> >> flagged this line of
> >>
> >>>> code as having a sign
> >>>> extension issue. He indicated that the correct
> >>>>
> >> way to implement
> >>
> >>>> this is to cast vm_stat.free_count to
> >>>> an unsigned 32-bit first, then cast the result
> >>>>
> >> to an unsigned
> >>
> >>>> long long. I'm still not sure of the correct
> >>>> coding and order of typecasting though since he
> >>>>
> >> didn't give an
> >>
> >>>> example.
> >>>>
> >>>> Tony
> >>>>
> >>>>
> >>>> On Jun 1, 2005, at 1:32 AM, William Kucharski
> >>>>
> >> wrote:
> >>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> On May 31, 2005, at 1:28 PM, Tony Scaminaci
> >>>>>
> >> wrote:
> >>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>> The following calculation of free memory (in
> >>>>>>
> >> bytes) works
> >>
> >>>>>> correctly on 32-bit platforms. On 64-bit
> >>>>>>
> >> systems (Tiger + G5),
> >>
> >>>>>> the result is a horrendously huge number
> rather
> >>>>>>
> >> than what is
> >>
> >>>>>> expected. I'm sure this is one of the
> >>>>>>
> >> conversion issues covered
> >>
> >>>>>> in the 64-bit transition guide (which I've
> read
> >>>>>>
> >> over multiple
> >>
> >>>>>> times already) but I can't seem to nail down
> >>>>>>
> >> the right
> >>
> >>>>>> way to do this calculation for both 32 and
> >>>>>>
> >> 64-bit systems.
> >>
> >>>>>>
> >>>>>> vm_size_t mac_pagesize;
> >>>>>> unsigned long long FreeMem; //
> >>>>>>
> >> Free real
> >>
> >>>>>> (physical) memory in bytes
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>> What's the "horrendously huge" number?
> >>>>>
> >>>>> The correct way to do this is to cast the
> number
> >>>>>
> >> of pages to an
> >>
> >>>>> unsigned long long and multiply it by
> >>>>>
> >> mac_pagesize, so the
> >>
> >>>>> initial code sample you had:
> >>>>>
> >>>>> FreeMem = ((unsigned long long)
> >>>>>
> >> vm_stat.free_count) *
> >>
> >>>>> mac_pagesize;
> >>>>>
> >>>>> should work just fine on all architectures.
> >>>>>
> >>>>> William Kucharski
> >>>>> email@hidden
> >>>>>
> >>
> >>
> >
>
>
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-kernel mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden