Filtering non-alphanumeric characters

[Christian Miller]

Hello. In an application I'm writing, I need to filter out non-
alphanumeric characters. Currently I loop through the text in the
editfield and manually filter A-Z, numbers, etc. This works great
for English. Can anyone share with me how I can accomplish the same
thing (automagically?) from within RB for non-English languages?

MacOS X 10.4.3, RB2005r4 IDE.

Thanks for the help,
Christian
Pariahware, Inc. Custom Software
<pariahware-qIiMK+***@public.gmane.org>
<http://www.pariahware.com>
--
God loved you so much that He gave His only son Jesus. What have you
done with God's gift?



_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

[Will Leshner]

Post by Christian Miller
In an application I'm writing, I need to filter out non-
alphanumeric characters.

What do you consider to be "non-alphanumeric"?

--
REALbasic news and tips: http://rbgazette.com
KidzMail & KidzBlog: http://haranbanjo.com


_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

[Theodore H. Smith]

Post by Will Leshner

Post by Christian Miller
In an application I'm writing, I need to filter out non-
alphanumeric characters.

What do you consider to be "non-alphanumeric"?

Usually alphanumeric is:

0-9
a-z
A-z
and anything with a codepoint above 127 :)

That's the simple way.

A more Unicode compatible way is more complex and really not so often
needed. I suppose you COULD filter out math, music and funny symbols
and non-English punctuation... It's usually not worth the time
investment to get this feature working.

_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

[Theodore H.Smith]

A-z

I meant:

A-Z

But you all knew that of course :)
_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

[Will Leshner]

Post by Theodore H. Smith
0-9
a-z
A-z
and anything with a codepoint above 127 :)

Then I wonder if Christian has merely to add a test for code points
above 127?

--
REALbasic news and tips: http://rbgazette.com
KidzMail & KidzBlog: http://haranbanjo.com


_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

[Joseph J. Strout]

Post by Theodore H. Smith
0-9
a-z
A-z
and anything with a codepoint above 127 :)

Then I wonder if Christian has merely to add a test for code points above 127?

Of course, users of other languages may object, since they consider
an accented letter to be alphanumeric -- which as I read it, was the
whole point of the original question.

Best,
- Joe

[Theodore H. Smith]

Post by Joseph J. Strout

Post by Will Leshner

Post by Theodore H. Smith
0-9
a-z
A-Z
and anything with a codepoint above 127 :)

Then I wonder if Christian has merely to add a test for code
points above 127?

Of course, users of other languages may object, since they consider
an accented letter to be alphanumeric -- which as I read it, was
the whole point of the original question.

An accented character has a code point of above 127, so yes my
original proposal does work for your example. My original reply also
gave examples which my proposal does not work for. Math symbols,
music. Actually even some English punctuation would not be stripped,
such as “ and ”.

A full Unicode compliant version is going to be much harder to write.
You'd end up needing a huge table of "if tests" in code, or some
other kind of data table.

Probably overkill, when > 127 does it for the most case.

_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

[Norman Palardy]

Post by Theodore H. Smith
Probably overkill, when > 127 does it for the most case.

But he needs it to work internationally as well


_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

[Christian Miller]

Post by Norman Palardy

Post by Theodore H. Smith
Probably overkill, when > 127 does it for the most case.

But he needs it to work internationally as well

Yes, that is the current issue. People speaking non-English
languages need to be able to select "filter non-alphanumeric
characters" and not have the application strip out any accented
characters.


Christian
Pariahware, Inc. Custom Software
<pariahware-qIiMK+***@public.gmane.org>
<http://www.pariahware.com>
--
God loved you so much that He gave His only son Jesus. What have you
done with God's gift?



_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

[Norman Palardy]

Post by Norman Palardy

Post by Theodore H. Smith
Probably overkill, when > 127 does it for the most case.

But he needs it to work internationally as well

Yes, that is the current issue. People speaking non-English languages
need to be able to select "filter non-alphanumeric characters" and not
have the application strip out any accented characters.

Does it have to be cross platform ?
The declares Charles suggested work fine on OS X
Not sure about Windows or Linux


_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

[Christian Miller]

Post by Norman Palardy

Post by Christian Miller
Yes, that is the current issue. People speaking non-English
languages need to be able to select "filter non-alphanumeric
characters" and not have the application strip out any accented
characters.

Does it have to be cross platform ?
The declares Charles suggested work fine on OS X
Not sure about Windows or Linux

Cross-platform is not necessary at this time (and possibly never). I
will take another look as I must have missed his message earlier today.


Christian
Pariahware, Inc. Custom Software
<pariahware-qIiMK+***@public.gmane.org>
<http://www.pariahware.com>
--
God loved you so much that He gave His only son Jesus. What have you
done with God's gift?



_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

[Joseph J. Strout]

Post by Joseph J. Strout

Post by Theodore H. Smith
0-9
a-z
A-z
and anything with a codepoint above 127 :)

Then I wonder if Christian has merely to add a test for code points above 127?

Of course, users of other languages may object, since they consider
an accented letter to be alphanumeric

Oops, I see that I read that wrong; code points above 127 were
suggested to be considered alphanumeric. But that doesn't work
either, as many code points outside the ASCII set aren't alphanumeric
at all, such as the bullet character, ellipsis character,
non-breaking space, currency symbols, and so on.

Best,
- Joe

[Phil M]

Post by Joseph J. Strout
Oops, I see that I read that wrong; code points above 127 were
suggested to be considered alphanumeric. But that doesn't work
either, as many code points outside the ASCII set aren't
alphanumeric at all, such as the bullet character, ellipsis
character, non-breaking space, currency symbols, and so on.

You would need to convert the characters into the Unicode code-points
(UTF16), and check to see if that code-point falls into the
acceptable range of Alphanumeric characters. But then you would also
need to decide if non-Latin character should be considered as
alphanumeric... such as Greek, Chinese etc.

Dim chars() As String = Split(ConvertEncoding(sourceString,
Encodings.UTF8), "")
Dim newChars() As String

For k As Integer = 0 To UBound(chars)
Select Case Asc(chars(k))
Case &h41 To &h5A, &h61 To &h7A, &hC0 To &hD6, &hD8 To &hF6,
&hF8 To &h0236, &h0250 To &h02AF, &h0386, &h0388 To &h047F, &h048A To
&h050F, &h1D00 To &h1FBC
newChars.Append(char)
End Select
Next

Return Join(newChars, "")

The code-points above include most of the Latin, Greek and Cyrillic
range. You can extend this into Armenian, Chinese, etc...

This is very rigid and does not include white space characters
(should it?).
_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

[Theodore H. Smith]

Post by Joseph J. Strout

Post by Joseph J. Strout

Post by Will Leshner

Post by Theodore H. Smith
0-9
a-z
A-Z
and anything with a codepoint above 127 :)

Then I wonder if Christian has merely to add a test for code
points above 127?

Of course, users of other languages may object, since they
consider an accented letter to be alphanumeric

Oops, I see that I read that wrong; code points above 127 were
suggested to be considered alphanumeric. But that doesn't work
either, as many code points outside the ASCII set aren't
alphanumeric at all, such as the bullet character, ellipsis
character, non-breaking space, currency symbols, and so on.

That was included in my point, yes. I actually did a few cases where
this approach doesn't give accurate results.


_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

[Christian Miller]

Post by Joseph J. Strout

Post by Will Leshner
Then I wonder if Christian has merely to add a test for code
points above 127?

Of course, users of other languages may object, since they consider
an accented letter to be alphanumeric -- which as I read it, was
the whole point of the original question.

Yes. :) I guess I need to know where to start filtering for non-
English languages.


Christian
Pariahware, Inc. Custom Software
<pariahware-qIiMK+***@public.gmane.org>
<http://www.pariahware.com>
--
God loved you so much that He gave His only son Jesus. What have you
done with God's gift?



_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

[Will Leshner]

Post by Christian Miller
Yes. :) I guess I need to know where to start filtering for non-
English languages.

I was afraid that was going to be the problem. This seems like a
really hard thing to do. How do you handle something like Japanese?

--
REALbasic news and tips: http://rbgazette.com
KidzMail & KidzBlog: http://haranbanjo.com


_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

[Thomas Benson]

I do this:

Custom Class with a Dictionary property "AllowedCharacters"

Then I create a series of functions in the subclass that append key
values (ie .value("a") = true) to the dictionary based on what target
language I am compiling for.
The dictionary is populated in the Open event by one of these
functions, which is called depends on a global value I set in
app.open which detects the platforms default language.

In the keydown event of the field, I simply put this one liner

If not me.allowedcharacters.haskey key then return true


It's fairly easy to accomplish the reverse too (ie a field witha
DisallowedCharacters dictionary).
It is a bit of manual work, but it makes localisation much easier.

- Tom

Post by Will Leshner

Post by Christian Miller
Yes. :) I guess I need to know where to start filtering for non-
English languages.

I was afraid that was going to be the problem. This seems like a
really hard thing to do. How do you handle something like Japanese?
--
REALbasic news and tips: http://rbgazette.com
KidzMail & KidzBlog: http://haranbanjo.com
_______________________________________________
<http://www.realsoftware.com/support/listmanager/>
<http://support.realsoftware.com/listarchives/lists.html>

_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

[Norman Palardy]

Post by Will Leshner

Post by Theodore H. Smith
0-9
a-z
A-z
and anything with a codepoint above 127 :)

Then I wonder if Christian has merely to add a test for code points
above 127?

But that would remove many useful characters in languages like French,
Danish, etc.


_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

[Ryan Marcus]

"In an application I'm writing, I need to filter out non-
alphanumeric characters. "

Sorry I could not reply with the whole message.. I never got it.

You can use the instr method with a loop

dim s as as string
dim i as integer

s = the string to test

while i<>len(s) + 1
if InStr(mid(s, i, 1), "abcdefghijklmnopqrstuvwxyz123456789") > 0 then
// The character mid(s, i, 1) is alpha-numeric.
else
// The character mid(s, i, 1) is not alpha numeric.
end if
wend



Its a tad but sloppy, but it should work. If you only need to test one
character at a time, you can do this:

if InStr(s, "abcdefghijklmnopqrstuvwxyz123456789") > 0 then
// True
else
// False
end if

Hope I could help.
--

Thanks, Ryan Marcus



_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

[dda]

Sounds like a job for a regex.

Dim r As New REgEx
r.SearchPattern="\W+"
r.ReplacementPattern=""
r.Options.ReplaceAllMatches=True
r.Options.Greedy=True
MyStringOnceFiltered=r.Replace(MyUnfilteredString)

\W catches everything outside [0-9a-zA-Z_]. If you have a more
specific definition of alphanumeric, we can surely work out something.

--
dda
libcurl4RB, [S]FTP transfers made easy
http://sungnyemun.org/?q=node/8

RBDeveloper Columnist, "Beyond the Limits"
http://rbdeveloper.com

Liste Française Solutions RB
http://www.solutionsrb.com/

Post by Christian Miller
Hello. In an application I'm writing, I need to filter out non-
alphanumeric characters. Currently I loop through the text in the
editfield and manually filter A-Z, numbers, etc. This works great
for English. Can anyone share with me how I can accomplish the same
thing (automagically?) from within RB for non-English languages?
MacOS X 10.4.3, RB2005r4 IDE.
Thanks for the help,
Christian
Pariahware, Inc. Custom Software
<http://www.pariahware.com>

[Charles Yeomans]

Post by Christian Miller
Hello. In an application I'm writing, I need to filter out
non-alphanumeric characters. Currently I loop through the text in the
editfield and manually filter A-Z, numbers, etc. This works great for
English. Can anyone share with me how I can accomplish the same thing
(automagically?) from within RB for non-English languages?
MacOS X 10.4.3, RB2005r4 IDE.

For MacOS X, you can call CFCharacterSet functions to do what you want.
Here is the code, which works in both PEF and Mach-O builds. Both
functions declared are available starting in MacOS 10.0.


Function IsAlphanumeric(character as String) as Boolean
Soft Declare Function CFCharacterSetGetPredefined Lib
"Carbon.framework" (theSetIdentifer as Integer) as Integer
Const kCFCharacterSetAlphaNumeric = 10

dim theRef as Integer =
CFCharacterSetGetPredefined(kCFCharacterSetAlphaNumeric)

Soft Declare Function CFCharacterSetIsCharacterMember Lib
"Carbon.framework" (theSet as Integer, theChar as Short) as Boolean

Return CFCharacterSetIsCharacterMember(theRef, Asc(character))
End Function

For other filtering, here are the remaining constants for predefined
sets.

Const kCFCharacterSetControl = 1
Const kCFCharacterSetWhitespace = 2
Const kCFCharacterSetWhitespaceAndNewline = 3
Const kCFCharacterSetDecimalDigit = 4
Const kCFCharacterSetLetter = 5
Const kCFCharacterSetLowercaseLetter = 6
Const kCFCharacterSetUppercaseLetter = 7
Const kCFCharacterSetNonBase = 8
Const kCFCharacterSetDecomposable = 9
Const kCFCharacterSetAlphaNumeric = 10
Const kCFCharacterSetPunctuation = 11
Const kCFCharacterSetIllegal = 12
Const kCFCharacterSetCapitalizedLetter =13
Const kCFCharacterSetSymbol = 14

--------------
Charles Yeomans

_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

[Christian Miller]

Post by Charles Yeomans
For MacOS X, you can call CFCharacterSet functions to do what you
want. Here is the code, which works in both PEF and Mach-O builds.
Both functions declared are available starting in MacOS 10.0.

...

Post by Charles Yeomans
dim theRef as Integer = CFCharacterSetGetPredefined
(kCFCharacterSetAlphaNumeric)

Thanks Charles! This looks like it will work. If I want to allow
whitespace, would I add the constants together, "or" them together,
or some other method?



Christian
Pariahware, Inc. Custom Software
<pariahware-qIiMK+***@public.gmane.org>
<http://www.pariahware.com>
--
God loved you so much that He gave His only son Jesus. What have you
done with God's gift?



_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

[Charles Yeomans]

Post by Christian Miller

Post by Charles Yeomans
For MacOS X, you can call CFCharacterSet functions to do what you
want. Here is the code, which works in both PEF and Mach-O builds.
Both functions declared are available starting in MacOS 10.0.

...

Post by Charles Yeomans
dim theRef as Integer =
CFCharacterSetGetPredefined(kCFCharacterSetAlphaNumeric)

Thanks Charles! This looks like it will work. If I want to allow
whitespace, would I add the constants together, "or" them together, or
some other method?

You need to use two separate function calls.

Function IsWhitespace(character as String) as Boolean
Soft Declare Function CFCharacterSetGetPredefined Lib
"Carbon.framework" (theSetIdentifer as Integer) as Integer
kCFCharacterSetWhitespace

dim theRef as Integer =
CFCharacterSetGetPredefined(kCFCharacterSetWhitespace)

Soft Declare Function CFCharacterSetIsCharacterMember Lib
"Carbon.framework" (theSet as Integer, theChar as Short) as Boolean

Return CFCharacterSetIsCharacterMember(theRef, Asc(character))
End Function

Then you would write

if IsAlphanumeric(theCharacter) or IsWhitespace(theCharacter) then
...

Usually you'd expect to be able to add or otherwise combine constants
only when the constants are powers of 2.

--------------
Charles Yeomans

_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

[G***@public.gmane.org]

Post by Will Leshner

Post by Christian Miller
In an application I'm writing, I need to filter out non-
alphanumeric characters.

What do you consider to be "non-alphanumeric"?

s="*9ab(cDF?>"
n=len(s)
t=""
if n > 0 then
for i = 1 to n
c = mid(s, i, 1) //the i_th chacter of s
if asc("a") <= asc( c) and asc(c) < asc("z") then t = t + c
if asc("A") <= asc( c) and asc(c) < asc("Z") then t = t + c
next
next

return t
_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

[Eric Baumgartner]

Post by Will Leshner

Post by Christian Miller
Yes. :) I guess I need to know where to start filtering for non-
English languages.

I was afraid that was going to be the problem. This seems like a
really hard thing to do. How do you handle something like Japanese?

Have a look at the Unicode Character Database.

http://www.unicode.org/ucd/

All the work of characterizing each code point has been done and
summed up in a single file.

http://www.unicode.org/Public/UNIDATA/UnicodeData.txt

Download the file, check the web site to figure out which fields and
properties you need to scan for...

http://www.unicode.org/Public/UNIDATA/UCD.html#UCD_File_Format
http://www.unicode.org/Public/UNIDATA/UCD.html#General_Category_Values

... and write a little script to extract the information you need.
Once you've got your master list of code points, you're just doing a
simple lookup.

(Or you could take Charles' advice and ask the OS to do it for you.)

[Theodore H. Smith]

Post by Eric Baumgartner

Post by Will Leshner

Post by Christian Miller
Yes. :) I guess I need to know where to start filtering for non-
English languages.

I was afraid that was going to be the problem. This seems like a
really hard thing to do. How do you handle something like Japanese?

Have a look at the Unicode Character Database.
http://www.unicode.org/ucd/
All the work of characterizing each code point has been done and
summed up in a single file.
http://www.unicode.org/Public/UNIDATA/UnicodeData.txt
Download the file, check the web site to figure out which fields
and properties you need to scan for...
http://www.unicode.org/Public/UNIDATA/UCD.html#UCD_File_Format
http://www.unicode.org/Public/UNIDATA/UCD.html#General_Category_Values
... and write a little script to extract the information you need.
Once you've got your master list of code points, you're just doing
a simple lookup.
(Or you could take Charles' advice and ask the OS to do it for you.)

That's exactly the sort of thing my ElfData plugin would perfectly
suited to doing.

You could do the filtering in one call to my MSR class. It would be:

FilteredString = msr.ReplaceAll( InputString )

You'd need to set up the MSR class via "msr.add From, To" and then
cache that MSR as a module property, so that you only set it up once
per app life-time. MSR is pretty smart, so if no filtering needs to
be done, it doesn't do any, it just returns the original string.

I've already written a UnicodeData.txt extractor (with my plugin)
that extracts decomposition data, lowercase data, uppercase data,
etc, from that file and a few other official files. It's my
UnicodeStuff.rb project available in my standard plugin download.

I could write a filter for non alpha-numeric characters, *if* someone
wanted it. I don't tend to write code unless there's a need for it.
Otherwise my work becomes aimless. It's often better to take a short
cut if it seems like it might do for now (like my > 127 trick), and
if it turns out to be a problem, then jump directly to doing it the
proper way.

By the way Eric, UnicodeData.txt doesn't contain all the unicode
data. There is some extra files needed for other text operations.


_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>