Repository navigation
Evaluate CLDR as database for cultures #128
Description
We're currently having various issues with the culture files generate from .NET (see label "cultures-bug"). We're considering moving to CLDR as the database for these files.
They're actually working on providing JSON files: http://unicode.org/cldr/trac/ticket/2733
We need to build a prototype to figure out if we can transform the CLDR data into something we can use here directly, or if we need to adapt the Globalize API to the CLDR data.
Activity
SlexAxton commentedon Oct 18, 2012
I know they don't love keeping it working (they do, it just is a hassle), but Dojo has scripts to take a lot of the CLDR info and turn it into usable JSON data and functions:
http://bugs.dojotoolkit.org/browser/dojo/util/trunk/buildscripts/cldr
It has some hefty pre-reqs, but that's to be expected with xlst and whatnot. I'd imagine you could do it a touch cleaner in node these days, but the core logic would be the same. I'm definitely looking into doing this, but I'd love for CLDR to just give us this stuff natively. :D
Adam Peller is the expert on this stuff and is/was active on that cldr ticket.
jzaefferer commentedon Oct 18, 2012
Thanks Alex, appreciate your input! More puzzle pieces to work with.
jzaefferer commentedon Oct 31, 2012
There's some activity on the cldr ticket: http://unicode.org/cldr/trac/ticket/2733#comment:22
Looks like for now our best bet is to just wait a few more weeks and hope for a release of those tools.
jzaefferer commentedon Jan 10, 2013
Btw. Tim Wood of Moment.js had general interest in using CLDR, but outlined some issues here: moment/moment#315 - he now also closed that ticket, since he doesn't have time to work on it. If we ever want to add relative time functions to Globalize, Moment.js would be a good starting point.
jzaefferer commentedon Mar 4, 2013
There's a Ldml2JsonConverter in CLDR now: http://unicode.org/cldr/trac/changeset/7886 Need to try that out.
jzaefferer commentedon Mar 25, 2013
Google's Closure library use CLDR as the datasource. I haven't yet figured out how they import the data, but the data itself is in this file: http://docs.closure-library.googlecode.com/git/closure_goog_i18n_datetimesymbols.js.source.html
The equivalent of our format method is in DateTimeFormat: http://docs.closure-library.googlecode.com/git/class_goog_i18n_DateTimeFormat.html
Source, using the above DateTimeSymbols: http://docs.closure-library.googlecode.com/git/closure_goog_i18n_datetimeformat.js.source.html
The source list the tokens it supports, which I can't find in the API document. The header says its based on CLDR standards: "Datetime formatting functions following the pattern specification as defined in JDK, ICU and CLDR, with minor modification for typical usage in JS. Pattern specification: (Refer to JDK/ICU/CLDR)"
jzaefferer commentedon Mar 25, 2013
Twitter published a JavaScript port of their Ruby CLDR wrapper 8 months ago: https://github.com/twitter/twitter-cldr-js
ericelliott commentedon May 12, 2013
CLDR JSON page and CLDR data in JSON format.
jzaefferer commentedon May 12, 2013
Thanks @dilvie for those links. Those look like it could just load them with
JSON.parse, then output the properties we actually need. Using the full CLDR files seems like a bad idea, since they are pretty massive.Its nice to see that there are properties for relative time formatting, e.g. "6 months ago". That's not something Globalize supports right now, but we could consider adding it.
ericelliott commentedon May 12, 2013
@jzaefferer Agreed. There's a lot of data that is not necessary most of the time. Maybe somebody could create a custom build script similar to http://projects.jga.me/jquery-builder/, so we get only the features we really need. That would actually be really nice in Globalize -- especially if all you need are string translations and number formatting. Or string translations and relative time (gentle nudge to anybody with more time than I can spare at the moment).
scottgonzalez commentedon May 13, 2013
It would be nice if we could just filter, as opposed to reformatting, the data. This would allow anyone who has full CLDR already available to just use that instead of duplicating some data with the Globalize files. We'll have to see if using the existing structure would become awkward.
andyearnshaw commentedon May 15, 2013
Hey guys, I noticed this issue earlier when I was investigating how to convert the CLDR data to JSON format. You need the CLDR core.zip and tools.zip files, then you need to use the Ldml2JsonConverter in the tools to convert the data.
You can use what I have so far as an example, in the tools folder at https://github.com/andyearnshaw/Intl.js.
ericelliott commentedon May 15, 2013
@andyearnshaw Thanks for the tips!
rxaviers commentedon Jul 4, 2013
What about coverage. Does CLDR data cover all culture data needed by Globalize? @jzaefferer have you or anyone looked into it already?
jzaefferer commentedon Jul 5, 2013
@rxaviers I don't think anyone did. Though Tim Wood, of moment.js, said that it lacks support for relative time, suggesting that everything else was there. We need to verify that either way.
19 remaining items
rxaviers commentedon Nov 1, 2013
@papandreou, our goal is to provide a set of tools that leverage the official CLDR JSON data. Check our in progress implementation on #172. We are using this library https://github.com/rxaviers/cldr to get help on CLDR data access.
Said that, converting XML data into the official JSON bindings is something we delegate to the official CLDR JSON tool.
If you find any issues or want to help us on this process you are welcome.
papandreou commentedon Nov 1, 2013
@rxaviers Yeah, I understood that. But since the JSON data is still incomplete, I just wanted to note that I've written a tool that extracts the data you need from the XML files.
rxaviers commentedon Nov 1, 2013
What's missing?
papandreou commentedon Nov 1, 2013
The current http://www.unicode.org/Public/cldr/24/json.zip only has 39 locales as opposed to 650+ in the XML data.
I don't know whether the data for the locales that are included is complete. It wasn't when I checked a few months ago.
rxaviers commentedon Nov 1, 2013
The available JSON data for download has the top 20 languages they (unicode.org CLDR staff) consider to be the "most used" languages. It contains the complete amount of data per language though. Also, they have been fully resolved.
You can use their official conversion tool (tools.zip) to generate the JSON representation of the languages not available in the ZIP. This ZIP contains a README with instructions on how to build the data. tools/scripts/CLDRWrapper may also be useful. Using the tool, you can opt to either generate resolved data, or unresolved to save space (or bandwidth) (-r false option of the conversion tool).
papandreou commentedon Nov 1, 2013
Oh, I missed that part. So json.zip is just a small sample of the real stuff. That's good news, I've been waiting for this to arrive :)
Thanks for the info.
rxaviers commentedon Nov 1, 2013
Yeap :). You are welcome. If you happen to find any flaws on the generated JSON, I will very much like want to know too. So, please let us know.
ragulka commentedon Nov 18, 2013
@rxaviers do you have any insight on how to actually use the conversion tool? i tried to follow the readme, but for someone who is not experienced with java, it's just too vague. I also posted the question on SO: http://stackoverflow.com/questions/20046099/how-to-build-json-data-from-cldr-data-using-the-java-conversion-tool
rxaviers commentedon Nov 21, 2013
@ragulka I completely understand your pain. I had the exact same issue as you had. Their README's instructions are currently misleading and should be fixed according to http://unicode.org/cldr/trac/ticket/6726.
rxaviers commentedon Dec 17, 2013
Closed by PR #172
Fixes globalizejs#128 by adding all locale attributes to attr_accessi…
Add :locale to attr_accessible list.
Merge branch 'attr_accessible_locale'