ICU integration and asset loading overhaul#37971

kg · 2020-06-16T15:48:16Z

This PR overhauls runtime startup/asset loading and adds support for ICU integration.

The mono-config.js format is reworked and simplified, with new functionality added:

Individual assets can be loaded from one or more remote sources with configurable fallback behavior
In addition to the existing support for loading assemblies, you can now pre-load arbitrary files into the native heap or into emscripten's virtual file system. VFS support previously only existed in runtime-test.js but now is available to any consumer of dotnet.js.
Assets can have a virtual path set so that their application-facing path does not necessarily have to match their path on the server.
One or more ICU data archives can be added to the assets list and will be automatically loaded and used to enable ICU-based globalization support.
Many configuration knobs that previously required API calls can now be set declaratively in the configuration file (environment variables, etc.)

WasmAppBuilder is updated to add ICUDataFiles and RemoteSources parameters that can be used to add the associated information to the config file declaratively from a msbuild project.

Various adjustments are made to existing tests and test cases so that they will pass with the addition of ICU integration.

ghost · 2020-06-16T15:48:23Z

Tagging subscribers to this area: @ViktorHofer
Notify danmosemsft if you want to be subscribed.

kg · 2020-06-16T15:49:02Z

Not sure what happened to runtime-test.js... does it have weird git line ending attributes set?

lewing · 2020-06-16T19:40:38Z

Not sure what happened to runtime-test.js... does it have weird git line ending attributes set?

Probably just .editorconfig fixing bad whitespace.

src/mono/configure.ac

src/mono/wasm/wasm.targets

Checkpoint Checkpoint Checkpoint Checkpoint Maybe fix whitespace damage Fix missing line endings in mono-config.js Maybe fix whitespace damage Add verification and fail-fast errors if ICU build data is missing or in the wrong place Checkpoint Checkpoint Checkpoint Checkpoint Checkpoint Checkpoint Checkpoint Fix whitespace damage Add editorconfig file for the wasm files Checkpoint Fix typo Checkpoint Checkpoint Cleanup Cleanup Reverse accidental removal of invariant default Checkpoint Sensible error when ICU_REPO is not set Checkpoint: Reset changes for mangled files Manually repair rebase damage These changes are no longer needed Repair rebase damage Fix mode damage Address PR feedback Fix icushim being built in static mode Remove broken wildcard dependency Fix sample startup

# Conflicts: # src/libraries/tests.proj

src/libraries/Native/Unix/System.Globalization.Native/pal_icushim_static.c

src/libraries/Common/src/Interop/Interop.TimeZoneInfo.Browser.cs

src/libraries/System.Globalization/tests/DateTimeFormatInfo/DateTimeFormatInfoTests.cs

src/libraries/System.Globalization/tests/System/Globalization/RegionInfoTests.cs

src/mono/netcore/sample/wasm/Program.cs

src/mono/netcore/System.Private.CoreLib/System.Private.CoreLib.csproj

eng/Version.Details.xml

src/libraries/Common/src/Interop/Interop.TimeZoneInfo.Browser.cs

src/libraries/Native/Unix/System.Globalization.Native/pal_icushim_static.c

…him_static.c Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com>

...braries/System.Private.CoreLib/src/System/TimeZoneInfo.Unix.TimeZoneDisplayName.Invariant.cs

safern · 2020-07-16T21:58:37Z

src/libraries/System.Runtime.Extensions/tests/System/AppDomainTests.cs

        }

        [Fact]
+        [PlatformSpecific(~TestPlatforms.Browser)]


Why do we need to disable this?

This PR overhauls runtime startup/asset loading and adds support for ICU integration. The mono-config.js format is reworked and simplified, with new functionality added: Individual assets can be loaded from one or more remote sources with configurable fallback behavior In addition to the existing support for loading assemblies, you can now pre-load arbitrary files into the native heap or into emscripten's virtual file system. VFS support previously only existed in runtime-test.js but now is available to any consumer of dotnet.js. Assets can have a virtual path set so that their application-facing path does not necessarily have to match their path on the server. One or more ICU data archives can be added to the assets list and will be automatically loaded and used to enable ICU-based globalization support. Many configuration knobs that previously required API calls can now be set declaratively in the configuration file (environment variables, etc.) WasmAppBuilder is updated to add ICUDataFiles and RemoteSources parameters that can be used to add the associated information to the config file declaratively from a msbuild project. Various adjustments are made to existing tests and test cases so that they will pass with the addition of ICU integration. Co-authored-by: EgorBo <egorbo@gmail.com> Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com> Co-authored-by: Larry Ewing <lewing@microsoft.com>

* ICU integration and asset loading overhaul (#37971) This PR overhauls runtime startup/asset loading and adds support for ICU integration. The mono-config.js format is reworked and simplified, with new functionality added: Individual assets can be loaded from one or more remote sources with configurable fallback behavior In addition to the existing support for loading assemblies, you can now pre-load arbitrary files into the native heap or into emscripten's virtual file system. VFS support previously only existed in runtime-test.js but now is available to any consumer of dotnet.js. Assets can have a virtual path set so that their application-facing path does not necessarily have to match their path on the server. One or more ICU data archives can be added to the assets list and will be automatically loaded and used to enable ICU-based globalization support. Many configuration knobs that previously required API calls can now be set declaratively in the configuration file (environment variables, etc.) WasmAppBuilder is updated to add ICUDataFiles and RemoteSources parameters that can be used to add the associated information to the config file declaratively from a msbuild project. Various adjustments are made to existing tests and test cases so that they will pass with the addition of ICU integration. Co-authored-by: EgorBo <egorbo@gmail.com> Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com> Co-authored-by: Larry Ewing <lewing@microsoft.com> * [wasm] Include data archives and timezone data by default (#39586) * Add data archive loading to the generic loading logic * [mono] Update ICU version, disable some tests for Browser (#39596) Co-authored-by: Katelyn Gadd <kg@luminance.org> Co-authored-by: EgorBo <egorbo@gmail.com> Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com>

kg added NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) NO-REVIEW Experimental/testing PR, do NOT review it labels Jun 16, 2020

Dotnet-GitSync-Bot added the area-Infrastructure label Jun 16, 2020

kg force-pushed the icu-loader branch from 2a8356a to d78ac2a Compare June 23, 2020 17:35

EgorBo reviewed Jun 25, 2020

View reviewed changes

src/mono/configure.ac Outdated Show resolved Hide resolved

EgorBo reviewed Jun 25, 2020

View reviewed changes

src/mono/wasm/wasm.targets Outdated Show resolved Hide resolved

kg force-pushed the icu-loader branch from bc4085f to 22976df Compare June 26, 2020 12:54

monojenkins mentioned this pull request Jun 26, 2020

[netcore] ICU integration mono/mono#20029

Merged

jaredpar mentioned this pull request Jun 26, 2020

OSX machines are de-provisioned during CI / PR runs leading to failures #34472

Closed

EgorBo mentioned this pull request Jun 29, 2020

[mono] Initial static ICU implementation for wasm #38057

Closed

kg force-pushed the icu-loader branch 2 times, most recently from ec2e896 to bcd4f0a Compare July 8, 2020 16:59

safern mentioned this pull request Jul 8, 2020

Builds timing out on CI #38964

Closed

EgorBo and others added 5 commits July 9, 2020 17:02

initial static ICU implementation for wasm

cbae5e5

Update pal_icushim_internal.h

f9fc4e3

Add stubs for tz and calendar, enable tests

b3e2325

Update System.Private.CoreLib.Shared.projitems

ca35363

kg force-pushed the icu-loader branch from bcd4f0a to ca35363 Compare July 10, 2020 00:43

EgorBo added 8 commits July 10, 2020 23:06

Merge branch 'master' of github.com:dotnet/runtime into icu-loader

0659c74

# Conflicts: # src/libraries/tests.proj

Fix EnumCalendarInfo for wasm (temp workaround)

72078b9

Merge branch 'master' of github.com:dotnet/runtime into icu-loader-egor2

04993e8

cccc

901c499

Fix small issues

bf6587a

drop ICU_REPO

39c315a

Update Program.cs

7c1855a

clean up

e2a57ff

tarekgh reviewed Jul 15, 2020

View reviewed changes

src/libraries/Native/Unix/System.Globalization.Native/pal_icushim_static.c Show resolved Hide resolved

lewing reviewed Jul 16, 2020

View reviewed changes

src/libraries/Common/src/Interop/Interop.TimeZoneInfo.Browser.cs Outdated Show resolved Hide resolved

lewing and others added 2 commits July 15, 2020 19:00

Update src/libraries/Common/src/Interop/Interop.TimeZoneInfo.Browser.cs

cafd0f2

Merge remote-tracking branch 'upstream/master' into icu-loader

c95ad39

marek-safar reviewed Jul 16, 2020

View reviewed changes

src/libraries/Common/src/Interop/Interop.TimeZoneInfo.Browser.cs Show resolved Hide resolved

marek-safar reviewed Jul 16, 2020

View reviewed changes

src/libraries/System.Globalization/tests/DateTimeFormatInfo/DateTimeFormatInfoTests.cs Outdated Show resolved Hide resolved

marek-safar reviewed Jul 16, 2020

View reviewed changes

src/libraries/System.Globalization/tests/System/Globalization/RegionInfoTests.cs Outdated Show resolved Hide resolved

marek-safar reviewed Jul 16, 2020

View reviewed changes

src/mono/netcore/sample/wasm/Program.cs Outdated Show resolved Hide resolved

marek-safar reviewed Jul 16, 2020

View reviewed changes

src/mono/netcore/System.Private.CoreLib/System.Private.CoreLib.csproj Show resolved Hide resolved

Revert sample changes

3eaa5fb

akoeplinger reviewed Jul 16, 2020

View reviewed changes

eng/Version.Details.xml Outdated Show resolved Hide resolved

src/libraries/Common/src/Interop/Interop.TimeZoneInfo.Browser.cs Show resolved Hide resolved

src/libraries/Native/Unix/System.Globalization.Native/pal_icushim_static.c Outdated Show resolved Hide resolved

kg and others added 7 commits July 16, 2020 03:47

Update src/libraries/Native/Unix/System.Globalization.Native/pal_icus…

794e290

…him_static.c Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com>

enable NativeCalendarName and RegionInfo.NativeName tests for wasm

34867d7

Update Version.Details.xml

fe0e686

bump ICU

9b20a78

Address feedback

49097be

Update Interop.TimeZoneInfo.Browser.cs

adc967e

Address feedback

6cf61e0

marek-safar reviewed Jul 16, 2020

View reviewed changes

...braries/System.Private.CoreLib/src/System/TimeZoneInfo.Unix.TimeZoneDisplayName.Invariant.cs Show resolved Hide resolved

EgorBo added 2 commits July 16, 2020 16:30

Address feedback

eff92b4

ignore Interop.TimeZoneInfo on wasm

7a1a28e

steveisok mentioned this pull request Jul 16, 2020

[Wasm] Tracking active test failures #38422

Closed

99 tasks

Address feedback

d64d256

lewing approved these changes Jul 16, 2020

View reviewed changes

safern reviewed Jul 16, 2020

View reviewed changes

safern approved these changes Jul 16, 2020

View reviewed changes

kg merged commit 5c99007 into dotnet:master Jul 16, 2020

ghost locked as resolved and limited conversation to collaborators Dec 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ICU integration and asset loading overhaul#37971

ICU integration and asset loading overhaul#37971
kg merged 68 commits intodotnet:masterfrom
kg:icu-loader

kg commented Jun 16, 2020 •

edited

Loading

Uh oh!

ghost commented Jun 16, 2020

Uh oh!

kg commented Jun 16, 2020

Uh oh!

lewing commented Jun 16, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

safern Jul 16, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Conversation

kg commented Jun 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Jun 16, 2020

Uh oh!

kg commented Jun 16, 2020

Uh oh!

lewing commented Jun 16, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

safern Jul 16, 2020

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

kg commented Jun 16, 2020 •

edited

Loading