Re: 64-bit problem with VM statistics

1 Jun 2005

      site_archiver@lists.apple.com
Delivered-To: darwin-kernel@lists.apple.com

This is the "memtest" utility - I'm the author for OS
X. It runs in single-user mode as well as mutlituser
mode and tests nearly all the installed ram in the
system (within about 97.2% of the currently free
amount as reported by vm_stat). Most people run it in
single-user mode so they can test virtually all of the
installed memory. I know some people at Apple are
aware of this app already because the algorithms have
found memory and cache problems where Apple's hardware
tests have not.

Under Panther, I couldn't malloc more than about 2 GB
of ram at one time. I chalked it up to some limitation
in malloc lib and did some heroics to prevent machines
from generating tons of malloc errors if they had more
than 2GB of installed ram. Going further and
attempting to mlock nearly all the free ram would
cause the kernel to become unresponsive, requiring a
hard reset. Over a long period of time and with a
bunch of testers, I found that if I limited the
allocation to 97.2% of the reported free memory in
single-user mode, the kernel would not lock up. I
don't know why this was the case but it simply was.
It's a well-known issue under both Panther and Jaguar
and I've been battling it for over a year. I even
filed a bug report in radar last year.

I've been getting increased requests to support up to
the full installed ram. Since Tiger's release, these
requests have gone out the roof because more people
are installing 4-8 GB in their Macs. They've been
complaining (loudly) that they couldn't test more than
2G of ram in one run under Panther but that limitation
magically disappeared under Tiger. My beta testers
found they could grab up to the 4G limit in both
single-user and multiuser modes but couldn't go beyond
that because the GetFreeMem() function was only
32-bit. So I changed it to be 64-bit clean or so I
thought. I've made all the changes I'm aware of to
fully support 64-bit mode but this one problem
remains. I've got over a dozen reports of G5's running
Tiger completely locking up if they try to test all of
their memory. I assume that malloc is trying to
allocate terrabytes of VM since the Avaialable Free
Memory number is so outrageously large. Whatever the
issue is, it's in this function and that function
determines how the rest of the app works or doesn't.

Memtest is available at
http://www.memtestosx.org/download if you want the
entire XCode2 project and it's licensed under GPL2. I
need to understand why the typecasting isn't working
correctly in the GetFreeMem function and fix it
quickly. I don't have a 64-bit platform to test on so
I'm relying on user feedback and testing. Whatever
help I can get is greatly appreciated.

Thanks,

Tony

--- Kevin Van Vechten <kevin@opendarwin.org> wrote:
...
Thanks, but I was hoping for a reference to the
project so we can all
be on the same page.  Are you looking in xnu
sources?  In
dynamic_pager sources in system_cmds?  Which file?
Could you elaborate how a large number can cause the
system to lock
up completely?  i.e. what's the actual operation in
question?
- Kevin
On Jun 1, 2005, at 1:03 PM, Tony Scaminaci wrote:
...
Kevin,
I've gotten feedback from a number of users that
the
large number being reported also causes their
systems
to lock up completely. It's not just a printf
problem.
Here's the source code for the function in
question:
unsigned long long GetFreeMem()
{
  vm_statistics_data_t        vm_stat;
  mach_port_t                host_priv_port,
host_port;
  mach_msg_type_number_t    host_count;
  kern_return_t                kern_error;
  unsigned long long        FreeMem;        //
Free real
(physical) memory
// Get host machine information
mach_port_t get_host_priv()
    {
      return(mach_host_self());
    }
mach_port_t get_host_port()
    {
      return(mach_host_self());
    }
// Get total system-wide memory usage structure
host_priv_port = get_host_priv();
  host_port = get_host_port();
  host_count = sizeof(vm_stat)/sizeof(integer_t);
  kern_error = host_statistics(host_priv_port,
HOST_VM_INFO, (host_info_t)&vm_stat, &host_count);
  if (kern_error != KERN_SUCCESS)
    {
      mach_error("host_info", kern_error);
      exit(EXIT_FAILURE);
    }
  host_page_size(host_port, &mac_pagesize);
    // Get
page size for this machine
  FreeMem = ((unsigned long long)
vm_stat.free_count)
* mac_pagesize;    // Calculate total free memory
in
bytes
  FreeMem &= 0xFFFFFFFFFFFFFFC0ull;
    // Align byte
count to 64-byte boundary
  return(FreeMem);
}
I need to have the correct number of bytes
returned to
the calling function in both 32-bit and 64-bit
modes.
This code works correctly in 32-bit environments
but
is breaking on 64-bit platforms.
Thanks,
Tony
--- Kevin Van Vechten <kevin@opendarwin.org>
wrote:
...
Perhaps the problem is with the format string
used
to print the
number instead of with the number itself?
Would someone mind pointing me to the specific
source file in question?
Thanks,
- Kevin
On Jun 1, 2005, at 6:23 AM, Tony Scaminaci wrote:
...
...
From: Tony Scaminaci
<friskythecat@sbcglobal.net>
Date: June 1, 2005 8:18:31 AM CDT
To: William Kucharski <kucharsk@mac.com>
Cc: Tony Scaminaci <friskythecat@sbcglobal.net>
Subject: Re: 64-bit problem with VM statistics
Unfortunately, this is the exact line that's
NOT
working on 64-bit
...
...
platforms. FreeMem is returning with the
following:
...
...
Available memory: 268267683840MB
(281299054850211840 bytes)
...
...
The number of bytes (which is what FreeMem is
holding) is way too
...
...
large. It should be around
4613734400 bytes on this particular machine
which
has about 4400
...
...
MB free as shown by top.
I received a bug post from John Caldwell who
flagged this line of
...
...
code as having a sign
extension issue. He indicated that the correct
way to implement
...
...
this is to cast vm_stat.free_count to
an unsigned 32-bit first,  then cast the result
to an unsigned
...
...
long long. I'm still not sure of the correct
coding and order of typecasting though since he
didn't give an
...
...
example.
Tony
On Jun 1, 2005, at 1:32 AM, William Kucharski
wrote:
...
...
...
On May 31, 2005, at 1:28 PM, Tony Scaminaci
wrote:
...
...
...
> The following calculation of free memory (in
>
bytes) works
...
...
...
> correctly on 32-bit platforms. On 64-bit
>
systems (Tiger + G5),
...
...
...
> the result is a horrendously huge number
rather
>
than what is
...
...
...
> expected. I'm sure this is one of the
>
conversion issues covered
...
...
...
> in the 64-bit transition guide (which I've
read
>
over multiple
...
...
...
> times already) but I can't seem to nail down
>
the right
...
...
...
> way to do this calculation for both 32 and
>
64-bit systems.
...
...
...
>
> vm_size_t                mac_pagesize;
> unsigned long long        FreeMem;        //
>
Free real
...
...
...
> (physical) memory in bytes
>
>
>
What's the "horrendously huge" number?
The correct way to do this is to cast the
number
of pages to an
...
...
...
unsigned long long and multiply it by
mac_pagesize, so the
...
...
...
initial code sample you had:
FreeMem = ((unsigned long long)
vm_stat.free_count) *
...
...
...
mac_pagesize;
should work just fine on all architectures.
William Kucharski
    kucharsk@mac.com
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-kernel mailing list      (Darwin-kernel@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.a...

This email sent to site_archiver@lists.apple.com

Tony Scaminaci

tags

participants (1)