Add optimized UTF-8 validation and transcoding apis, hook them up to UTF8Encoding (#21948)

* Add optimized UTF-8 validation and transcoding logic
- Hook it up through the existing Utf8 public static APIs
- Move some shared methods out of ASCIIUtility
- Hook it up through the Utf8String ctor

* Hook up new UTF-8 logic through UTF8Encoding
- Add vectorized UTF-16 validation and transcoded byte counts
- Move Utf16Utility into Unicode namespace alongside Utf8Utility
- Fix some bugs in DecoderNLS's draining logic

* Improve perf of "is ASCII?" inner loop in UTF-8 validation.

* Remove SSE41.X64 optimization from AsciiUtility
RyuJIT now handles this optimally

* Clarify that vector read is unaligned

* Simplify vectorized logic; remove unnecessary adjustment

* PR feedback: GetElement(0) -> Sse2.StoreLow

* PR feedback
- Simplify CountNumberOfLeadingAsciiBytesFrom24BitInteger
- Extract some consts out to top of file w/ comments

* PR feedback: Enable SSE2 in Utf16Utility code

* Expand masks in Utf8Utility, fix const in fallback path

* Temporarily disable failing CoreFX tests

* Fix incorrect Debug.Assert statements

* Add comments tracking JIT workarounds.

* Rename DWORD -> UInt32 throughout API surface

* Re-flow Utf8Utility.Helpers

* PR feedback: Fix typos

* PR feedback: CountNumberOfLeadingAsciiBytesFrom24BitInteger

* PR feedback: Remove redundant endianess checks

* PR feedback: Validate nint definitions

* PR feedback: Clarify charIsNonAscii vector usage

* PR feedback: document tempUtf8CodeUnitCountAdjustment usage

* Fix compilation failure in Utf16Utility

* PR feedback: Clarify 3-byte sequence processing

* Add missing check to 3-byte processing logic

* Clarify comment in 3-byte processing

1 files changed, 11 insertions, 3 deletions

diff --git a/tests/CoreFX/CoreFX.issues.json b/tests/CoreFX/CoreFX.issues.json
index 9104e2f260..ac2c24277d 100644
--- a/tests/CoreFX/CoreFX.issues.json
+++ b/tests/CoreFX/CoreFX.issues.json
@@ -896,15 +896,23 @@
             "methods": [
                 {
                     "name": "System.Text.Tests.EncoderConvert2.EncoderASCIIConvertMixedASCIIUnicodeCharArrayPartial",
-                    "reason": "https://github.com/dotnet/coreclr/issues/23020"
+                    "reason": "https://github.com/dotnet/coreclr/issues/23864"
                 },
                 {
                     "name": "System.Text.Tests.EncoderConvert2.EncoderUTF8ConvertMixedASCIIUnicodeCharArrayPartial",
-                    "reason": "https://github.com/dotnet/coreclr/issues/23020"
+                    "reason": "https://github.com/dotnet/coreclr/issues/23864"
                 },
                 {
                     "name": "System.Text.Tests.EncoderConvert2.EncoderUTF8ConvertUnicodeCharArrayPartial",
-                    "reason": "https://github.com/dotnet/coreclr/issues/23020"
+                    "reason": "https://github.com/dotnet/coreclr/issues/23864"
+                },
+                {
+                    "name": "System.Text.Tests.NegativeEncodingTests.GetByteCount_Invalid",
+                    "reason": "https://github.com/dotnet/coreclr/issues/23864"
+                },
+                {
+                    "name": "System.Text.Tests.UTF8EncodingDecode.Decode_InvalidBytes",
+                    "reason": "https://github.com/dotnet/coreclr/issues/23864"
                 }
             ]
         }