Add Span support in tokenizer's Model abstraction

[tarekgh]

This change is adding Span support to the Tokenizer's Model abstraction. This will help in performance and not restrict us in the future to use spans without to worry about more memory allocations.

[tarekgh]

CC @ericstj @luisquintanilla @michaelgsharp

[stephentoub]

Thanks!

[codecov]

Codecov Report

Attention: Patch coverage is 68.89764% with 79 lines in your changes are missing coverage. Please review.

Project coverage is 68.80%. Comparing base (d0aa2c2) to head (40a669f).
Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7035      +/-   ##
==========================================
+ Coverage   68.79%   68.80%   +0.01%     
==========================================
  Files        1254     1255       +1     
  Lines      250204   250248      +44     
  Branches    25529    25533       +4     
==========================================
+ Hits       172127   172190      +63     
+ Misses      71467    71449      -18     
+ Partials     6610     6609       -1     

Flag	Coverage Δ
Debug	68.80% <68.89%> (+0.01%)	⬆️
production	63.24% <68.65%> (+0.01%)	⬆️
test	88.50% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
src/Microsoft.ML.Tokenizers/Model/Cache.cs	96.66% <100.00%> (+48.72%)	⬆️
src/Microsoft.ML.Tokenizers/Tokenizer.cs	88.37% <100.00%> (ø)
...crosoft.ML.Tokenizers/Utils/Helpers.netstandard.cs	81.25% <100.00%> (+6.25%)	⬆️
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs	86.86% <100.00%> (ø)
...crosoft.ML.Tokenizers.Tests/EnglishRobertaTests.cs	95.68% <100.00%> (ø)
src/Microsoft.ML.Tokenizers/Model/Model.cs	38.46% <0.00%> (ø)
...rc/Microsoft.ML.Tokenizers/Model/EnglishRoberta.cs	79.63% <93.47%> (+0.95%)	⬆️
...rosoft.ML.Tokenizers/Utils/StringSpanOrdinalKey.cs	82.14% <82.14%> (ø)
src/Microsoft.ML.Tokenizers/Utils/LruCache.cs	73.68% <59.45%> (+7.01%)	⬆️
src/Microsoft.ML.Tokenizers/Model/Tiktoken.cs	62.17% <51.28%> (+0.80%)	⬆️
... and 1 more

... and 3 files with indirect coverage changes

[stephentoub]

Thanks.