Re: Accessibility and UIInspector to find text under cursor
Re: Accessibility and UIInspector to find text under cursor
- Subject: Re: Accessibility and UIInspector to find text under cursor
- From: Gary Yuen <email@hidden>
- Date: Thu, 24 Aug 2006 01:25:43 -0700
Hi Bill,
Thank you very much again.
I've been trying to find more documentation for accessibility. As far
as I can tell, there's only the HIServices docs and release notes for
parameterized attributes. Maybe there's more but I haven't been able
to find it.
I also haven't had much luck with AXValueCreate. It fails with a
Carbon point and an NSPoint. Maybe it doesn't acccept Carbon points
and the NSPoint I'm passing it may not be within the window bounds.
My luck hasn't been too great so far.
It also seems that WebKit doesn't provide the necessary parameterized
attributes to get word or character under the mouse.
I've discovered that TSM Document Access is another method. I'm not
sure if there are others. Getting into that or another method may be
complex enough it could take me some time to figure out. I wish there
were more examples or documentation beyond framework docs.
I'd like to continue this though I may put it on hold for now. My
efforts to read the docs (and finding the relevant ones and making
sense of them) have been kind of slow.
Thanks Bill.
Gary
在 2006-8-17，上午2:20，Bill Cheeseman 写道：
on 2006-08-16 9:30 PM, Gary Yuen at email@hidden wrote:
I've looked into this further and am not sure if Accessibility can
give me the specific word or character under the mouse. While using
UI Browser and TextEdit, I see that some attributes for the text area
are range for position and string for range. Do you know if it's
possible to pass the mouse position to get the specific character
under the mouse?
You need to find the documentation for the parameterized attribute
accessibility functions, where all this is explained. Here's a
summary:
Only one of the parameterized attributes takes a 'position'
parameter, so
when you're reading the screen you have to start with that
attribute: range
for position. Before you can use it, you must, of course, get the
current
location of the mouse on the screen using standard Carbon or Cocoa
techniques. The location is expressed as a point value.
This attribute returns the character range of a single glyph based
on its
position on the screen. The returned value is the character range
of the
glyph under the mouse. In a screen reader application, use the
resulting
character range returned by this attribute as a route into the other
parameterized methods which take a character index, character
range, or
(indirectly from the character index) line index as input. The
location
field of the character range is the character index of the first
character
of the glyph, and the length field of the character range is the
number of
characters to the next glyph. (It is important to respect glyph
boundaries.)
You can always get the UI element under the mouse using one of the
other
accessibility functions, AXUIElementCopyElementAtPosition. I assume
you've
figured that out already.
So: Start by getting the current mouse position. Then use that
position as
input to get the range for position of the glyph under the mouse
within the
text storage contained in the UI element under the mouse, which
includes
both the starting point of the glyph under the mouse expressed as
an index
from the beginning of the entire text storage containing that
glyph, and the
length of the glyph.
From that point on, it's up to you to parse the text in the
vicinity of that
glyph. One way -- which is what UI Browser does, for convenience --
is to
get the range of the line of text in which the glyph is located.
Using the
line range, you can obtain the text of the line, then break it into
words
and find the word that contains the glyph. To do all this, you just
need to
keep working off of the index that you obtained from the range for
position
attribute, which includes the index of the glyph under the mouse.
If AXUIElementCopyParameterizedAttributeValue is able to do this, I'm
not sure what format the third parameter (parameter) needs to be in.
The 2nd one (parameterized attribute) one seems to take constant like
kAXRangeForPositionParameterizedAttribute and
kAXStringForRangeParameterizedAttribute.
This is all explained in the documentation. The third parameter,
when you're
getting range for position, is the position of the mouse expressed
as a
point value. In the accessibility API, when you're dealing with
point and
range values, you have to use the accessibility AXValueCreate
function,
which is well documented. Here's some Cocoa code from UI Browser
that may
help you to figure this out:
- (NSValue *)AXRangeForPosition:(NSValue *)glyphPosition {
NSPoint point = [glyphPosition pointValue];
CFTypeRef parameterValue = AXValueCreate(kAXValueCGPointType,
&point);
CFTypeRef value = [(NSValue *)[self
objectForAttribute:kAXRangeForPositionParameterizedAttribute
parameter:parameterValue] autorelease];
if (parameterValue) CFRelease(parameterValue);
CFRange decodedValue;
if (AXValueGetValue(value, kAXValueCFRangeType, &decodedValue)) {
return [NSValue valueWithRange:NSMakeRange
(decodedValue.location,
decodedValue.length)];
}
return nil;
}
Where the method -objectForAttribute:parameter: is coded like this
(leaving
out some details if an error occurs):
- (CFTypeRef)objectForAttribute:(CFStringRef)attribute
parameter:(CFTypeRef)parameter {
CFTypeRef value;
AXError err = AXUIElementCopyParameterizedAttributeValue([self
elementRef], attribute, parameter, &value);
if (err == kAXErrorSuccess) return value;
// error handling code goes here
}
I'm most interested in text with Safari. When using the UI Browser
with Safari, I don't see any attributes that would let me specific
text more fine-grained than the entire area.
Safari is difficult to parse, but only because it breaks the
typical HTML
page up into dozens or hundreds of groups, each of which might
contain text
or something else. Once you've found a group that contains text, the
parameterized attributes work on that text the same way they work
on any
other text.
You can always find the UI element under the mouse by using the basic
accessibility screen reader function,
AXUIElementCopyElementAtPosition. Then
use the kAXPositionAttribute attribute of the returned element to
get its
position on the screen, if needed.
I hope you find this helpful. There is documentation available, and
you
really need to read it and understand it before you do any coding
in the
accessibility API.
--
Bill Cheeseman - email@hidden
Quechee Software, Quechee, Vermont, USA
http://www.quecheesoftware.com
PreFab Software - http://www.prefab.com/scripting.html
The AppleScript Sourcebook - http://www.AppleScriptSourcebook.com
Vermont Recipes - http://www.stepwise.com/Articles/VermontRecipes
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Accessibility-dev mailing list (Accessibility-
email@hidden)
Help/Unsubscribe/Update your Subscription:
40mac.com
This email sent to email@hidden
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Accessibility-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden