Fwd: Migrations encoding
Fwd: Migrations encoding
- Subject: Fwd: Migrations encoding
- From: email@hidden
- Date: Sun, 10 Oct 2010 06:15:00 -0700
- Resent-date: Sun, 10 Oct 2010 16:23:12 +0300
- Resent-from: Mahdi Mankai <email@hidden>
- Resent-message-id: <email@hidden>
- Resent-to: email@hidden
Hi,
I am running into the same issue below.
I am using ERXJDBCUtilities.executeUpdateScriptFromResourceNamed() to insert few rows in the DB. The DB is using UTF-8 Encoding. File containing the SQL statements is using UTF-8 as well. The problem is that a value like "Québec" is not inserted properly using the ERXJDBCUtilities.executeUpdateScriptFromResourceNamed(). The same fine inserts the data properly if executed directly using Postgres command line.
I tried to modify the file.encoding Property with no luck.
I tried the UTF-16 encoding: I change the file encoding (made sure to tell Eclipse about the UTF-16 encoding). Now, the migration fails ERXJDBCUtilities.executeUpdateScriptFromResourceNamed() throws an exception:
Caused by: java.lang.RuntimeException: Failed to execute 'ˇ˛INSERT INTO entity_status(status_name) values ('ACTIVE')'.
at er.extensions.jdbc.ERXJDBCUtilities.executeUpdateScript(ERXJDBCUtilities.java:660)
at er.extensions.jdbc.ERXJDBCUtilities.executeUpdateScript(ERXJDBCUtilities.java:614)
at er.extensions.jdbc.ERXJDBCUtilities.executeUpdateScriptFromResourceNamed(ERXJDBCUtilities.java:734)
at ca.fusiondev.migration.Peripheral0.upgrade(Peripheral0.java:581)
at er.extensions.migration.ERXMigrationDatabase$Migration.upgrade(ERXMigrationDatabase.java:444)
at er.extensions.migration.ERXMigrator$ERXMigrationAction.doPerform(ERXMigrator.java:473)
... 16 more
Caused by: org.postgresql.util.PSQLException: ERROR: insufficient data left in message
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:299)
at er.extensions.jdbc.ERXJDBCUtilities.executeUpdateScript(ERXJDBCUtilities.java:656)
... 21 more
However setting the JVM parameter file.encoding=UTF-8 in the command line solves my issue and data are inserted properly.
I am wondering if somebody knows about a better solution for this issue.
Thanks,
Mahdi
Begin forwarded message:
> From: Johann Werner <email@hidden>
> Date: July 20, 2009 1:02:17 PM GMT+03:00
> To: WebObjects-Dev Mailing List List <email@hidden>
> Subject: Re: Migrations encoding
>
> Hi,
>
> I just came across the same problem. I have an UTF-8 encoded file that should be executed during migrations. On my development machine everything works as expected but on the deployment machine unicode characters get garbled. After some debugging I found the difference between both machines (development is 10.5.7 Client, deployment 10.5.7 Server): the java system property file.encoding is UTF-8 on my development machine and on the server it is set to MacRoman. I don't know why this setting is different nor how I can change this default.
>
> By adding a -Dfile.encoding=UTF-8 to the JVM arguments everything is ok but this means that either I have to add this in JavaMonitor or add it to the generated WOA-startup script. Both things you can easily forget and screw up your migrations. So another solution I tried is to add
>
> System.setProperty("file.encoding", "UTF-8"); (1)
>
> to the application constructor. This seems to remedy the situation but unfortunately the server showed the same garbled string :( By stepping through the code the line
>
> return new String(ERXFileUtilities.bytesFromInputStream(in)); (2)
>
> in ERXStringUtilities.stringFromInputStream calls
>
> String->StringCoding.decode->Converters.getDefaultEncodingName
>
> which returns again MacRoman. Could it be that setting the system property (1) does not alter the value globally? How are you assuring that the file encoding defaults to UTF-8 in your WO applications? Is there a special trick? Is changing the startup script in the ant build the way to go (so should it possibly be made the default in WOLips)?
>
> A temporary patch would be to change the line (2) to
>
> return new String(ERXFileUtilities.bytesFromInputStream(in), Charset.defaultCharset());
>
> jw
>
>
> Am 15.06.2009 um 20:45 schrieb Tusker:
>
>> Yup. I made sure it was using bbedit. I read in the sql file manually(using FileInputStream with UTF-16 encoding) and inserted it into the database via editingcontext with no problem at all. How else can I make sure it's encoded properly?
>>
>>
>> On Jun 14, 2009, at 12:20 PM, Chuck Hill wrote:
>>
>>>
>>> On Jun 12, 2009, at 10:59 AM, Tusker wrote:
>>>
>>>> Hi,
>>>>
>>>> Same problem. I'm actually using a *.sql file within the Resources folder. It's been working great until I got to unicode characters.
>>>>
>>>> I'm using this to call the *.sql. I've made sure that it is encoded (UTF-16). Also tried (UTF-8)
>>>
>>>
>>> You made sure it was encoded in UTF-16 or you just told Eclipse that it was? I don't think that Eclipse will re-encode the file if you do that.
>>>
>>>
>>> Chuck
>>>
>>>
>>>>
>>>> ERXJDBCUtilities.executeUpdateScriptFromResourceNamed(channel, "mysqltest.sql", null);
>>>>
>>>> I tried changing the name to "name". Works fine until I introduce unicode characters.
>>>>
>>>> Somewhere between reading the file to execution, the encoding is not right.
>>>>
>>>> This is what ends up in the database:
>>>>
>>>> Belgium/België/Belgique
>>>>
>>>> Thanks
>>>>
>>>>
>>>> On Jun 12, 2009, at 10:34 AM, Jon Nolan wrote:
>>>>
>>>>> Tusker wrote:
>>>>>> I changed the file to be UTF-16 but I get an error now. I use the same sql file and manually run it in frontbase, it works. Example of the statement:
>>>>>>
>>>>>> update Country set name = 'Belgium/België/Belgique' where country_id = 2;
>>>>>>
>>>>> "name" is a reserved word in Frontbase although FBManager (not sure about sql92) can often/usually/always handle non-quoted reserved words depending on how you're using them. I've always had to double-quote reserved words when working in Migrations though. i.e.
>>>>>
>>>>> ERXJDBCUtilities.executeUpdate(database.adaptorChannel(), "update SITE set \"TYPE\" = 1 where \"TYPE\" = 10");
>>>>>
>>>>>
>>>>> Of course it might still be an encoding issue but let's rule out this potential problem first. Try..
>>>>>
>>>>>
>>>>> update Country set "name" = 'Belgium/België/Belgique' where country_id = 2;
>>>>> _______________________________________________
>>>>> Do not post admin requests to the list. They will be ignored.
>>>>> Webobjects-dev mailing list (email@hidden)
>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>
>>>>> This email sent to email@hidden
>>>>
>>>> _______________________________________________
>>>> Do not post admin requests to the list. They will be ignored.
>>>> Webobjects-dev mailing list (email@hidden)
>>>> Help/Unsubscribe/Update your Subscription:
>>>>
>>>> This email sent to email@hidden
>>>
>>> --
>>> Chuck Hill Senior Consultant / VP Development
>>>
>>> Come to WOWODC'09 in San Fran this June!
>>> http://www.wocommunity.org/wowodc09/
>>>
>
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Webobjects-dev mailing list (email@hidden)
> Help/Unsubscribe/Update your Subscription:
>
> This email sent to email@hidden
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden