Cqrlog-improved errors during import

2 posts / 0 new
Last post
SP2L
SP2L's picture
Cqrlog-improved errors during import

Hello Saku.

I noticed something new when importing my latest ADIF file downloaded from QRZ.com into CQRLOG2.
The import contains all 34010 QSOs, so no records are lost, but the error log contains many messages such as:

name error -> Андрей Русяев fix -> ______ ______
qth error -> Бишкек fix -> ______

and:

qth error -> São Paulo ,SP fix -> S_o Paulo ,SP

followed by:

ERROR: UTF8 characters in tag data, not ADIF standard! Modifying AdifData
There are also messages about fields being shortened, for example:
qsl_via shrink to 30 chrs: QSL VIA DIRECT MORE US $ 2 + SASE LOTW

I have not changed the CQRLOG2 binary since our previous investigation.

Is this UTF-8 handling expected behaviour in CQRLOG2, or is this something that should be changed in the importer? The ADIF was downloaded directly from QRZ.com and contains the original UTF-8 characters.

Tom - SP2L

oh1kh
Cqrlog-improved errors during import

HI Tom!

Study https://adif.org/317/ADIF_317.htm#QSO_Fields
You will see that for example NAME and QTH have alternative adif tages NAME_INTL and QTH_INTL

Tag NAME should carry only 7bit ASCII, I.E. letters A-Z. No ÖÄÅöäåóé etc.
Instead NAME_INTL can carry all UTF8 charactes.

How ever NAME_INTL should only be used with .adx files. Because of that limitation almost all programs now are pushing UTF8 to NAME and QTH and nobody cares that.

What is the use of standards if nobody do not follow them?

Cqrlog does not import or export UTF8 to NAME or QTH. Import will drop(replace) chars over $7F
How ever also Cqrlog beaks standards by using NAME_INTL and QTH_INTL for import and export with .adi files. That is less bad, and Cqrlog needs it because backups are made with adif and they do not restore similar without UTF8 in some tags like NAME and QTH.

IMHO:
Adif.org should allow "_INTL" tags also to .adi files. That would fix most problems if just programmers then use them instead.

Problem arises when you get length of tag. if you assume it is standard 7bit ASCII then one byte is one letter and function to use is length().
That results fine <NAME:4>SAKU string length is then 4 as bytes, as tag says.
But if there is UTF8 chars: <NAME:4>SÄKU then string length is 5 as bytes and tag says 4

Try to use "sed" and change tags <NAME and <QTH to <NAME_INTL and <QTH_INTL and try what happens with import then.
Then Cqrlog knows to use function UTF8length() instead of length() when it compares length given in tag to real byte count of tag string.

I am not sure if it works(too long since last looking at import source), but is easy and fast to try.

-----------

Shrinks happen because the database field length is less than the string length trying to import to it.
It is just for giving info to user instead that silently importing them and later user would notice that database filed (column in log) dos not carry all information.
This is giving user the opportunity to save the important part of too long text.

--
Saku
OH1KH