Allow slugs to contain most utf8 characters

Summary:
Ref T2632. Fixes T1466.

Currently, we normalize slugs (and thus Phriction URIs and canonical project names) to a small number of latin characters. Instead, blacklist a few characters and permit everything else (including utf8 characters).

When generating Phriction URIs, encode any utf8 characters. This means we render URIs encoded, but browsers handle this fine and display them readably in the URI and address bar, etc.

The blacklisted characters are mostly for practical reasons: \x00-\x19 are control characters, `#%?` are meaningful in URIs, `+` is sometimes configured to be interprted as space by apache, etc., `<>\\` are just silly, `&= ` are largely cosmetic.

This allows some silly stuff, like generating URIs with zero-width spaces and RTL markers in them. Possibly we should go blacklist those characters at some point.

Depends on: D5191

Test Plan: {F34402}

Reviewers: AnhNhan, chad, vrana

Reviewed By: chad

CC: aran

Maniphest Tasks: T1466, T2632

Differential Revision: https://secure.phabricator.com/D5192

src/applications/phriction/storage/PhrictionDocument.php

@@ -42,6 +42,10 @@ final class PhrictionDocument extends PhrictionDAO
     if ($slug == '/') {
       return $prefix;
     } else {
+      // NOTE: The effect here is to escape non-latin characters, since modern
+      // browsers deal with escaped UTF8 characters in a reasonable way (showing
+      // the user a readable URI) but older programs may not.
+      $slug = phutil_escape_uri($slug);
       return $prefix.$slug;
     }
   }

src/infrastructure/util/PhabricatorSlug.php

@@ -3,14 +3,11 @@
 final class PhabricatorSlug {
 
   public static function normalize($slug) {
-
-    // TODO: We need to deal with unicode at some point, this is just a very
-    // basic proof-of-concept implementation.
-
-    $slug = strtolower($slug);
     $slug = preg_replace('@/+@', '/', $slug);
     $slug = trim($slug, '/');
-    $slug = preg_replace('@[^a-z0-9/]+@', '_', $slug);
+    $slug = phutil_utf8_strtolower($slug);
+    $slug = preg_replace("@[\\x00-\\x19#%&+=\\\\?<> ]+@", '_', $slug);
+    $slug = preg_replace('@_+@', '_', $slug);
     $slug = trim($slug, '_');
 
     return $slug.'/';
@@ -20,8 +17,8 @@ final class PhabricatorSlug {
     $parts = explode('/', trim($slug, '/'));
     $default_title = end($parts);
     $default_title = str_replace('_', ' ', $default_title);
-    $default_title = ucwords($default_title);
-    $default_title = nonempty($default_title, 'Untitled Document');
+    $default_title = phutil_utf8_ucwords($default_title);
+    $default_title = nonempty($default_title, pht('Untitled Document'));
     return $default_title;
   }
 

src/infrastructure/util/__tests__/PhabricatorSlugTestCase.php

@@ -13,8 +13,10 @@ final class PhabricatorSlugTestCase extends PhabricatorTestCase {
       'derp//derp'        => 'derp/derp/',
       'DERP//DERP'        => 'derp/derp/',
       'a B c'             => 'a_b_c/',
-      '-1~2.3abcd'        => '1_2_3abcd/',
-      "T\x95O\x95D\x95O"  => 't_o_d_o/',
+      '-1~2.3abcd'        => '-1~2.3abcd/',
+      "T\x00O\x00D\x00O"  => "t_o_d_o/",
+      'x#%&+=\\?<> y'     => 'x_y/',
+      "\xE2\x98\x83"      => "\xE2\x98\x83/",
     );
 
     foreach ($slugs as $slug => $normal) {