testing WIP-pyshapelib-Unicode

[Bram de Greve]

Bernhard Reiter wrote:
> On Thursday 03 January 2008 02:07, Bram de Greve wrote:
>   
>> - when creating a new shapefile in Thuban, it now automatically will
>> create an UTF-8 encoded DBF file.  You will see this by an accompanying
>> .CPG file.  This is necessary, as the UTF-8 encoding cannot be set by
>> the LDID field in the DBF file itself.  ArcView should understand this
>> .CPG file though (they're the ones using it ;)
>>     
>
> I wonder what should be the default.
>   

The user should be able to configure this in Thuban.  The default
probably should be LDID/87 (= 0x57).  That's an ANSI code page.

> Is it possible to switch to old behaviour?
>   

Sure.  I merely choose UTF-8 in the Unicode branch because I'm testing
UTF-8 at the moment.  We can switch back to any other behaviour in
Thuban/Model/table.py

> Are ut-8 dbf files standard with shapefiles now?
>   

No, a default installation will use Windows ANSI.

There's a catch though.  Windows ANSI is not uniquely defined.  It could
be any of the CP125x pages, whatever is set by the box.  At least that's
what I think.  I'm not sure about it, an expert would need to confirm
this.  At any rate, wherever Windows ANSI or just ANSI is mentioned, I
assume CP1252, as that's the most common mentioned ANSI page.  (0x3 and
0x57 are both set to CP1252 in pyshapelib).

>   
>> - all strings read from shapefiles are now passed to Thuban as unicode
>> strings.  But when viewing shapefiles with exotic UTF-8 content (and I
>> mean _exotic_ ;) things bork.  Be warned ...
>>     
>
> What does "bork" mean in this context?
> We probably should take at least a little effort to not make Thuban break 
> completely when it encounters strange stuff, so where is the problem coming 
> from?
>   

"to bork" means that Thuban is not completely prepared to handle exotic
unicode characters.    Thuban/Model/table.py now returns unicode strings
exclusively, and I don't think that's where the problem is caused.  When
I try to display the table, things go really wrong.  I must prepare a
new nice test shapefile so you can see yourself.  I tried this a month
ago, but I've lost the testfile =)

I'll do this later today.

Bramz

> Bernhard
>
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> Thuban-devel mailing list
> Thuban-devel at intevation.de
> https://intevation.de/mailman/listinfo/thuban-devel
>