Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skolt Sámi tokenisation completely broken #96

Open
snomos opened this issue Oct 2, 2024 · 0 comments
Open

Skolt Sámi tokenisation completely broken #96

snomos opened this issue Oct 2, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@snomos
Copy link
Member

snomos commented Oct 2, 2024

The following text is behaving well in most Mac apps using the MacDivvun speller:

Sääʹm-musikkneǩ Anna Lumikivi ǩeâšttââll tän ǩiiđ Voice of Finland -tv-prograammâst. Lumikivi piâzzi mieʹldd prograamm vuõssmõs Ääni ratkaisee -vuässa, koʹst ǩeâšttõõlli peäʹsse juåtkku, jõs ǩeâlb haʹrjjtõʹtti stuuʹl jåårǥle ǩeâšttõõlli läullmõõžž ääiʹj. Televisio musikkprograamm lie pâi leämmaž peäglva Lumikivi piârrjest Aanar Keväjääuʹrest. – Mââimõs vuâra, ko ǩiõʹččem prograamm muu päärnaivuiʹm, smiõʹttem, što kâʹl toʹben õõlǥči kuullâd sääʹmǩiõl še. Teʹl pueʹđi juurd, što ââʹn mon tuejjääm mâid-ne aivv jõll da piijam ooccmõõžž tok - da de mon pieʹssem tok, mušttal Lumikivi

Sääʹmǩiõll puätt koll’jed tän ǩiiđ peäglvaž The Voice of Finland -tv-prograammâst Anna Lumikivi haaʹlad čuäʹjted nuõrid, što säämas kaʹnnat raajjâd musiikk. Lääʹddjânnam jõnn musikk-ǩeâlbain jeät leäkku ni kuäʹss ouddâl kuullâm nuõrttsääʹmǩiõllsaž laulli.

It looks like this, with misspellings and Finnish words underlined:

SMS text as screen sitot, with misspellings underlined

Running the same text through either of the GDocs of Office 365 extensions returns the following tokens flagged as misspellings:

Bilde 2 10 2024 klokken 19 06

It should have been one token, not two.

@snomos snomos added the bug Something isn't working label Oct 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant