fix #556 correct non-english hashtags #557

vivganes · 2023-05-14T19:42:08Z

vivganes commented

2023-05-14 19:42:08 +00:00

What was the problem?
Our code was taking only the first hashtag. We need to take all the hashtags from the content

How did I verify the fix?
I created a new event with similar text as the one in the issue #556 , using the new code.

New Event ID: note1hrkkcwrlkm8hq0a2rfl5tksng62pug8arnn5qf27c0syg5gv0pcs2je6n9

New Event JSON

{
  "content": "Retesting international hashtags in snort. Please ignore.\n\n\n#ಕನ್ನಡ is in Kannada\n#தமிழ் is in Tamil\n#сообщество is in Russian\n#漫画 is in Japanese\n#समाज is in Hindi",
  "created_at": 1684091894,
  "id": "b8ed6c387fb6cf703faa1a7f45da1346941e20fd1ce740255ec3e044510c7871",
  "kind": 1,
  "pubkey": "facdaf1ce758bdf04cdf1a1fa32a3564a608d4abc2481a286ffc178f86953ef0",
  "sig": "b1721d7bd60e899905406b1ce95eb60abd4ce75768ff5d93b9dfea3d8b3741699d592d15e678dc28db0c44e88b88442119f49137aa28ad4715a60061286924b7",
  "tags": [
    [
      "t",
      "ಕನ್ನಡ"
    ],
    [
      "t",
      "தமிழ்"
    ],
    [
      "t",
      "сообщество"
    ],
    [
      "t",
      "漫画"
    ],
    [
      "t",
      "समाज"
    ]
  ]
}

Screenshot of JSON Diff

**What was the problem?** Our code was taking only the first hashtag. We need to take all the hashtags from the content **How did I verify the fix?** I created a new event with similar text as the one in the issue #556 , using the new code. New Event ID: `note1hrkkcwrlkm8hq0a2rfl5tksng62pug8arnn5qf27c0syg5gv0pcs2je6n9` **New Event JSON** ```json { "content": "Retesting international hashtags in snort. Please ignore.\n\n\n#ಕನ್ನಡ is in Kannada\n#தமிழ் is in Tamil\n#сообщество is in Russian\n#漫画 is in Japanese\n#समाज is in Hindi", "created_at": 1684091894, "id": "b8ed6c387fb6cf703faa1a7f45da1346941e20fd1ce740255ec3e044510c7871", "kind": 1, "pubkey": "facdaf1ce758bdf04cdf1a1fa32a3564a608d4abc2481a286ffc178f86953ef0", "sig": "b1721d7bd60e899905406b1ce95eb60abd4ce75768ff5d93b9dfea3d8b3741699d592d15e678dc28db0c44e88b88442119f49137aa28ad4715a60061286924b7", "tags": [ [ "t", "ಕನ್ನಡ" ], [ "t", "தமிழ்" ], [ "t", "сообщество" ], [ "t", "漫画" ], [ "t", "समाज" ] ] } ``` **Screenshot of JSON Diff** ![image](/attachments/b92c4647-b291-4c9f-b841-de561b7cfad8)

image.png

54 KiB

vivganes added 1 commit 2023-05-14 19:42:09 +00:00

fix #556 944bbf9ed5

Kieran reviewed 2023-05-14 21:14:17 +00:00

packages/app/src/System/EventBuilder.ts Outdated

						
				@ -48,0 +46,4 @@

				        this.#replaceMention(m)

				      );

				      const hashTags = [...this.#content.matchAll(new RegExp(HashtagRegex, "g"))];

Kieran commented

2023-05-14 21:14:06 +00:00

Maybe you can just add the /g to the regex?

Maybe you can just add the `/g` to the regex?

vivganes commented

2023-05-15 00:03:31 +00:00

Done!

vivganes marked this conversation as resolved

vivganes added 1 commit 2023-05-15 00:03:06 +00:00

fix review comment 270c4b2f01

Kieran reviewed 2023-05-15 08:41:55 +00:00

packages/app/src/System/EventBuilder.ts

						
				@ -45,3 +45,1 @@

				      this.#content = this.#content

				        .replace(/@n(pub|profile|event|ote|addr|)1[acdefghjklmnpqrstuvwxyz023456789]+/g, m => this.#replaceMention(m))

				        .replace(HashtagRegex, m => this.#replaceHashtag(m));

				      this.#content = this.#content.replace(/@n(pub|profile|event|ote|addr|)1[acdefghjklmnpqrstuvwxyz023456789]+/g, m =>

Kieran commented

2023-05-15 08:41:55 +00:00

Does this work then? Seems like there is a flaw here which also might affect mentions?

vivganes commented

2023-05-15 14:25:49 +00:00

Yes this seems to work without issues.

Just to ensure that I am not affecting the mentions, I tested with this note

https://snort.social/e/note190vnpkk5p3e9fq6xmpewr5kdm808esxdfymrqptzk7at00meuhyq2ekf03

The event JSON turned out to be as expected


{
  "content": "Retesting tagging users and hashtags. Please ignore\n\nnostr:npub1cz2ve34nk0ukn0ph4yq2qx3ud8rfy5e0ak4epx42dn8gha0sdgpsgra9kv is the first mention\n\nnostr:npub1ltx67888tz7lqnxlrg06x234vjnq349tcfyp52r0lstclp548mcqnuz40t is the second mention\n\n\n#ಕನ್ನಡ\n#漫画",
  "created_at": 1684160601,
  "id": "2bd930dad40c72548346d872e1d2cdd9de7cc0cd4936300562b7bab7bf79e5c8",
  "kind": 1,
  "pubkey": "39ef11ee6d8e80d361c0938372c4edae87ffa4fdb55285302e4a8dbedaeac7b3",
  "sig": "3c345883fa0a55c4b3e767e45029c8a33e1c943d8bb9222e94070f93a626910d82a4c857303511a58e8a7fc81ac9cd12cb2647d1fee23a7f06ae046d6006f6e8",
  "tags": [
    [
      "p",
      "c094ccc6b3b3f969bc37a900a01a3c69c692532fedab909aaa6cce8bf5f06a03"
    ],
    [
      "p",
      "facdaf1ce758bdf04cdf1a1fa32a3564a608d4abc2481a286ffc178f86953ef0"
    ],
    [
      "t",
      "ಕನ್ನಡ"
    ],
    [
      "t",
      "漫画"
    ]
  ]
}

The similar note, when tested on snort.social without this fix gives the following faulty JSON for the note https://snort.social/e/note134xyhwstlavjv644pafs7x0pkvlg0qa2sflx582nkd4wr9um0mwshth3s2

{
  "content": "Testing tagging users and hashtags. Please ignore\n\nnostr:npub1cz2ve34nk0ukn0ph4yq2qx3ud8rfy5e0ak4epx42dn8gha0sdgpsgra9kv   \n\nnostr:npub1ltx67888tz7lqnxlrg06x234vjnq349tcfyp52r0lstclp548mcqnuz40t \n\n#ಕನ್ನಡ \n#漫画",
  "created_at": 1684160398,
  "id": "8d4c4bba0bff59266ab50f530f19e1b33e8783aa827e6a1d53b36ae1979b7edd",
  "kind": 1,
  "pubkey": "39ef11ee6d8e80d361c0938372c4edae87ffa4fdb55285302e4a8dbedaeac7b3",
  "sig": "d4f784e94494cbe45578c4ed57dd678524ade06f6687151a9a3e304850deb49e9221b85cc58733ef45daa43c2bae52bdf6e45889b75033fe8e2e2b7201ff38c2",
  "tags": [
    [
      "p",
      "c094ccc6b3b3f969bc37a900a01a3c69c692532fedab909aaa6cce8bf5f06a03"
    ],
    [
      "p",
      "facdaf1ce758bdf04cdf1a1fa32a3564a608d4abc2481a286ffc178f86953ef0"
    ],
    [
      "t",
      "ಕನ್ನಡ"
    ]
  ]
}

Yes this seems to work without issues. Just to ensure that I am not affecting the mentions, I tested with this note https://snort.social/e/note190vnpkk5p3e9fq6xmpewr5kdm808esxdfymrqptzk7at00meuhyq2ekf03 The event JSON turned out to be as expected ``` { "content": "Retesting tagging users and hashtags. Please ignore\n\nnostr:npub1cz2ve34nk0ukn0ph4yq2qx3ud8rfy5e0ak4epx42dn8gha0sdgpsgra9kv is the first mention\n\nnostr:npub1ltx67888tz7lqnxlrg06x234vjnq349tcfyp52r0lstclp548mcqnuz40t is the second mention\n\n\n#ಕನ್ನಡ\n#漫画", "created_at": 1684160601, "id": "2bd930dad40c72548346d872e1d2cdd9de7cc0cd4936300562b7bab7bf79e5c8", "kind": 1, "pubkey": "39ef11ee6d8e80d361c0938372c4edae87ffa4fdb55285302e4a8dbedaeac7b3", "sig": "3c345883fa0a55c4b3e767e45029c8a33e1c943d8bb9222e94070f93a626910d82a4c857303511a58e8a7fc81ac9cd12cb2647d1fee23a7f06ae046d6006f6e8", "tags": [ [ "p", "c094ccc6b3b3f969bc37a900a01a3c69c692532fedab909aaa6cce8bf5f06a03" ], [ "p", "facdaf1ce758bdf04cdf1a1fa32a3564a608d4abc2481a286ffc178f86953ef0" ], [ "t", "ಕನ್ನಡ" ], [ "t", "漫画" ] ] } ``` The similar note, when tested on snort.social without this fix gives the following faulty JSON for the note https://snort.social/e/note134xyhwstlavjv644pafs7x0pkvlg0qa2sflx582nkd4wr9um0mwshth3s2 ``` { "content": "Testing tagging users and hashtags. Please ignore\n\nnostr:npub1cz2ve34nk0ukn0ph4yq2qx3ud8rfy5e0ak4epx42dn8gha0sdgpsgra9kv \n\nnostr:npub1ltx67888tz7lqnxlrg06x234vjnq349tcfyp52r0lstclp548mcqnuz40t \n\n#ಕನ್ನಡ \n#漫画", "created_at": 1684160398, "id": "8d4c4bba0bff59266ab50f530f19e1b33e8783aa827e6a1d53b36ae1979b7edd", "kind": 1, "pubkey": "39ef11ee6d8e80d361c0938372c4edae87ffa4fdb55285302e4a8dbedaeac7b3", "sig": "d4f784e94494cbe45578c4ed57dd678524ade06f6687151a9a3e304850deb49e9221b85cc58733ef45daa43c2bae52bdf6e45889b75033fe8e2e2b7201ff38c2", "tags": [ [ "p", "c094ccc6b3b3f969bc37a900a01a3c69c692532fedab909aaa6cce8bf5f06a03" ], [ "p", "facdaf1ce758bdf04cdf1a1fa32a3564a608d4abc2481a286ffc178f86953ef0" ], [ "t", "ಕನ್ನಡ" ] ] } ```

vivganes marked this conversation as resolved

vivganes added 35 commits 2023-05-15 14:33:35 +00:00

chore: update lang 084983c0b6

setup drone build

continuous-integration/drone/push Build encountered an error

Details

21057e72fc

chore: change pipeline names

continuous-integration/drone/push Build is failing

Details

a3cba728a9

chore: update crowdin config