No announcement yet.

Problem with umlauts when importing CSV to DSF

  • Filter
  • Time
  • Show
Clear All
new posts

  • Problem with umlauts when importing CSV to DSF

    I've set up a Directsmile VDP item in DSF and tried to use batch processing using a CSV file. Unfortunately, regardless of using "Unicode" or "Western Europe", any special characters (like umlauts) are unreadable in the job preview. Although the preview of the imported data is correct.
    If I enter the values in single record mode, everything's fine as well as during preview in VDP Studio...

    I guess DSF has some problems handing over the data to DSM?

    UPDATE: It also fails when using a DSF datasource (without uploading a CSV). So it seems to be a problem in the way, batch data is handed over to DSM.

    UPDATE 2: Okay, it doesn't seem to be directly DSF related. When I use a CSV as database in VDP Studio's local datainterface, it also shows this encoding bug (but it works when importing an xls file).

    UPDATE 3: Next one: I've examined the way, the data arrives at the backend using the DSMI Server Monitor. It looks like DSF sends batch request with unescaped fieldValues (e.g. fieldValue="äöü") whereas single request are made using fieldValue=[[{{urlDecode("%30a...")}}]].

    UPDATE 4: Okay, until my customer finally manages to open a support ticket, I'll continue to do this on my own. It seems to me that the encoding of the ticket values switches from unicode to utf-8 at a certain point but DSMG still expects unicode. Therefore I had to do a very ugly hack to overcome this. I need to find and replace every possible utf-8 character in every field using this vbScript function:
    function UnicodeToAscii(pstrUnicode)
          Dim llngLength
          Dim llngIndex
          Dim llngAscii
          Dim lstrAscii
          llngLength = Len(pstrUnicode)
          For llngIndex = 1 To llngLength
                llngAscii = Asc(Mid(pstrUnicode, llngIndex, 1))
                If llngAscii = 195 Then
                    llngAscii = Asc(Mid(pstrUnicode, llngIndex+1, 1))
                    Select Case llngAscii
                        Case 164
                            lstrUnicode = lstrUnicode & "ä"
                        Case 132
                            lstrUnicode = lstrUnicode & "Ä"
                        Case 188
                            lstrUnicode = lstrUnicode & "ü"
                        Case 156
                            lstrUnicode = lstrUnicode & "Ü"
                        Case 182
                            lstrUnicode = lstrUnicode & "ö"
                        Case 150
                            lstrUnicode = lstrUnicode & "Ö"
                        Case 159
                            lstrUnicode = lstrUnicode & "ß"
                        Case Else
                            lstrUnicode = lstrUnicode & "?"
                    End Select
                    llngIndex = llngIndex + 1
                    lstrUnicode = lstrUnicode & Mid(pstrUnicode, llngIndex, 1)
                End If
          UnicodeToAscii = lstrUnicode
    End function
    Now at least german umlauts get printed correctly, although I have to use this function for every single field. I really hope, EFI addresses this within the next years ;-)
    Last edited by kopierschnitte; 26th November 2017, 03:40.