Hello,
I have application running on centos 3.8 which brings back some data
from a MS SQL server db, and writes it to disk.

However there are some special characters ( u with 2 dots overhead,
for example)  in the data which appear as ? in the linux file created.
I am told the database uses CP 1252, which means the u with 2 dots
overhead,is character 252,

The output of locale is;
$ locale
LANG=C
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL
What can I do to fix this problem please?

Am Donnerstag 24 September 2009 11:57, syd_p a écrit :

Based on the example I mentioned using LANG=de would be
a possible solution.

No, the default CTYPE for de is ISO-8859-1.

CP1252 is a superset of ISO-8859-1. The accented letters are the same.

Based on the example I mentioned using LANG=de would be
a possible solution.

But we are seeing French, Spanish and German "special"
characters which are supported by MS's CP 1252.

→ printf '“„�\n' | iconv -tlatin1 | iconv -flatin1
iconv: Séquence d'échappement illégale à la position 0
→ printf '“„�\n' | iconv -tcp1252 | iconv -fcp1252
“„�

Not quite sure how you did the printf above tho.

And not quite sure what I should set to say LANG
and LC_ALL to en_us first and check that out?

I did not originally set up the box (actually there are 6 or 8
of them) but I think that LANG=C was done cos there was
a problem with LANG-en_us.

Am Dienstag, 29. September 2009 10:31, syd_p a écrit :

But with LANG=C which I thought was only 7 bits the following
printfs work just fine.

$ printf "(octal 353) is the character \0353\n"
(octal 353) is the character ë
printf "(octal 361) is the character \0361\n"
(octal 361) is the character ñ

Good, the typeface (font) has the characters you need.

These are two of the characters in the MSSQL db which the
application (not open source) handles as "?".

Check if the application is really the cause of the problem.
For file 'foo', generated by application, run :
LANG=en_US.ISO-8859-15 tr -d '\000-\177' <foo | od -b

--
printf -v email $(echo \ 155 141 162 143 145 154 142 162 165 151 \
156 163 155 141 100 171 141 150 157 157 056 143 157 155|tr \  \\\\)
#  Live every life as if it were your last!  #
$ cat foo

Am Dienstag, 29. September 2009 10:31, syd_p a écrit :

But with LANG=C which I thought was only 7 bits the following
printfs work just fine.

$ printf "(octal 353) is the character \0353\n"
(octal 353) is the character ë
printf "(octal 361) is the character \0361\n"
(octal 361) is the character ñ

Good, the typeface (font) has the characters you need.

These are two of the characters in the MSSQL db which the
application (not open source) handles as "?".

Check if the application is really the cause of the problem.
For file 'foo', generated by application, run :
LANG=en_US.ISO-8859-15 tr -d '\000-\177' <foo | od -b

--
printf -v email $(echo \ 155 141 162 143 145 154 142 162 165 151 \
156 163 155 141 100 171 141 150 157 157 056 143 157 155|tr \  \\\\)
#  Live every life as if it were your last!  #
$ cat foo

On 29 Sep, 10:37, Marcel Bruinsma <m... at (no spam) nomail.afraid.org> wrote:

Am Dienstag, 29. September 2009 10:31, syd_p a écrit :

But with LANG=C which I thought was only 7 bits the following
printfs work just fine.

$ printf "(octal 353) is the character \0353\n"
(octal 353) is the character ë
printf "(octal 361) is the character \0361\n"
(octal 361) is the character ñ

Good, the typeface (font) has the characters you need.

These are two of the characters in the MSSQL db which the
application (not open source) handles as "?".

Check if the application is really the cause of the problem.
For file 'foo', generated by application, run :
LANG=en_US.ISO-8859-15 tr -d '\000-\177' <foo | od -b

--
printf -v email $(echo \ 155 141 162 143 145 154 142 162 165 151 \
156 163 155 141 100 171 141 150 157 157 056 143 157 155|tr \  \\\\)
#  Live every life as if it were your last!  #

$ cat  foo
A?O
$  od -b  foo
0000000 101 077 117 012
0000004
octal 101 =  A
octal 077 =  ?
octal 077 =  O
middle char should be capital ñ

$ LANG=en_US.ISO-8859-15 tr -d '\000-\177' <foo | od -b
0000000
not quite sure what this does
Ah yes I am - it deletes all "normal" chars and passes the remainder

I entered the commands as suggested
LANG=en_US.CP1252 locale -> Bad
LANG=en_US.ISO-8859-15 locale -> Good
LANG=en_US.ISO-8859-1 locale -> Good
LANG=en_US.UTF-8 locale -> Good

On 29 Sep, 12:43, syd_p <sydneypue... at (no spam) yahoo.com> wrote:

On 29 Sep, 10:37, Marcel Bruinsma <m... at (no spam) nomail.afraid.org> wrote:

Am Dienstag, 29. September 2009 10:31, syd_p a écrit :

But with LANG=C which I thought was only 7 bits the following
printfs work just fine.

$ printf "(octal 353) is the character \0353\n"
(octal 353) is the character ë
printf "(octal 361) is the character \0361\n"
(octal 361) is the character ñ

Good, the typeface (font) has the characters you need.

These are two of the characters in the MSSQL db which the
application (not open source) handles as "?".

Check if the application is really the cause of the problem.
For file 'foo', generated by application, run :
LANG=en_US.ISO-8859-15 tr -d '\000-\177' <foo | od -b

--
printf -v email $(echo \ 155 141 162 143 145 154 142 162 165 151 \
156 163 155 141 100 171 141 150 157 157 056 143 157 155|tr \  \\\\)
#  Live every life as if it were your last!  #

$ cat  foo
A?O
$  od -b  foo
0000000 101 077 117 012
0000004
octal 101 =  A
octal 077 =  ?
octal 077 =  O
middle char should be capital ñ

$ LANG=en_US.ISO-8859-15 tr -d '\000-\177' <foo | od -b
0000000
not quite sure what this does ;-)

Ah yes I am - it deletes all "normal" chars and passes the remainder
to od...
not quite sure what all zeros as the out means tho...

And on a Centos 5.3 box I just invoked
$  printf "(octal 353) is the character \0353\n"
(octal 353) is the character 3
on the centos 3.8 box I got the expected output of ë
Ahh - I see need to specify the hex value thusly:

But with LANG=C which I thought was only 7 bits the following
printfs work just fine.

$ printf "(octal 353) is the character \0353\n"
(octal 353) is the character ë
printf "(octal 361) is the character \0361\n"
(octal 361) is the character ñ

These are two of the characters in the MSSQL db which the
application (not open source) handles as "?".

On 29 Sep, 12:43, syd_p <sydneypue... at (no spam) yahoo.com> wrote:

On 29 Sep, 10:37, Marcel Bruinsma <m... at (no spam) nomail.afraid.org> wrote:

Am Dienstag, 29. September 2009 10:31, syd_p a écrit :

But with LANG=C which I thought was only 7 bits the following
printfs work just fine.

$ printf "(octal 353) is the character \0353\n"
(octal 353) is the character ë
printf "(octal 361) is the character \0361\n"
(octal 361) is the character ñ

Good, the typeface (font) has the characters you need.

These are two of the characters in the MSSQL db which the
application (not open source) handles as "?".

Check if the application is really the cause of the problem.
For file 'foo', generated by application, run :
LANG=en_US.ISO-8859-15 tr -d '\000-\177' <foo | od -b

--
printf -v email $(echo \ 155 141 162 143 145 154 142 162 165 151 \
156 163 155 141 100 171 141 150 157 157 056 143 157 155|tr \  \\\\)
#  Live every life as if it were your last!  #

$ cat  foo
A?O
$  od -b  foo
0000000 101 077 117 012
0000004
octal 101 =  A
octal 077 =  ?
octal 077 =  O
middle char should be capital ñ

$ LANG=en_US.ISO-8859-15 tr -d '\000-\177' <foo | od -b
0000000
not quite sure what this does ;-)

Ah yes I am - it deletes all "normal" chars and passes the remainder
to od...
not quite sure what all zeros as the out means tho...

And on a Centos 5.3 box I just invoked
$  printf "(octal 353) is the character \0353\n"
(octal 353) is the character 3
on the centos 3.8 box I got the expected output of ë
Ahh - I see need to specify the hex value thusly:

$  od -b  foo
0000000 101 077 117 012
0000004
octal 101 =  A
octal 077 =  ?
octal 077 =  O
middle char should be capital ñ

$ LANG=en_US.ISO-8859-15 tr -d '\000-\177' <foo | od -b
0000000
not quite sure what this does
Ah yes I am - it deletes all "normal" chars and passes
the remainder to od...

not quite sure what all zeros as the out means tho...

And on a Centos 5.3 box I just invoked
$ printf "(octal 353) is the character \0353\n"
(octal 353) is the character 3

on the centos 3.8 box I got the expected output of ë

$ printf "(hex EB) is the character \xEB\n"
(hex EB) is the character ë
this works on 3.8 and 5.3

Am Dienstag, 29. September 2009 15:54, syd_p a écrit :

$ printf "(hex EB) is the character \xEB\n"
(hex EB) is the character ë
this works on 3.8 and 5.3

This a nice one to try. It shows the terminal mapping:
printf '\xa4 \x80\n'

To understand that:
zgrep -E '\x(80|a4)' /usr/share/i18n/charmaps/ISO-8859-15.gz
zgrep -E '\x(80|a4)' /usr/share/i18n/charmaps/ISO-8859-1.gz
zgrep -E '\x(80|a4)' /usr/share/i18n/charmaps/CP1252.gz

--
printf -v email $(echo \ 155 141 162 143 145 154 142 162 165 151 \
156 163 155 141 100 171 141 150 157 157 056 143 157 155|tr \  \\\\)
#  Live every life as if it were your last!  #
Finally I have had some success with: