Pre-processing - Revision history

Tamchyna at 10:03, 11 March 2015

2015-03-11T10:03:42Z

← Older revision		Revision as of 10:03, 11 March 2015
Line 101:		Line 101:

	It is essential for data tokenization to be '''consistent'''. All of our training data should conform to the same pre-processing scheme and an identical pipeline should be applied at test time (when our system runs and we translate new data).		It is essential for data tokenization to be '''consistent'''. All of our training data should conform to the same pre-processing scheme and an identical pipeline should be applied at test time (when our system runs and we translate new data).

			== Related Material ==

			[https://www.youtube.com/watch?v=Lx4eD9HcGI0 Data Cleaning and Tokenization (Moses tutorial)]

	== Exercises ==		== Exercises ==

Tamchyna at 15:22, 27 January 2015

2015-01-27T15:22:12Z

← Older revision		Revision as of 15:22, 27 January 2015
Line 3:		Line 3:
	\|image = [[File:bear-with-us.png\|200px]]		\|image = [[File:bear-with-us.png\|200px]]
	\|label1 = Lecture video:		\|label1 = Lecture video:
	\|data1 = [http://example.com web '''TODO'''] <br/> [~~http~~://www.youtube.com/watch?v=GDij7urWeOk Youtube]		\|data1 = [http://example.com web '''TODO'''] <br/> [https://www.youtube.com/watch?v=GDij7urWeOk&list=PLpiLOsNLsfmbeH-b865BwfH15W0sat02V&index=3 Youtube]
	\|label2 = Exercises:		\|label2 = Exercises:
	\|data2 = [https://codex3.ms.mff.cuni.cz/codex-trans/?groupId=3&taskId=6&module=groups%2Ftasks&page=specification Lowercasing] <br/> [https://codex3.ms.mff.cuni.cz/codex-trans/?groupId=3&taskId=7&module=groups%2Ftasks&page=specification Deaccenting]		\|data2 = [https://codex3.ms.mff.cuni.cz/codex-trans/?groupId=3&taskId=6&module=groups%2Ftasks&page=specification Lowercasing] <br/> [https://codex3.ms.mff.cuni.cz/codex-trans/?groupId=3&taskId=7&module=groups%2Ftasks&page=specification Deaccenting]
	}}		}}

	{{#ev:youtube\|GDij7urWeOk\|800\|center}}		{{#ev:youtube\|https://www.youtube.com/watch?v=GDij7urWeOk&list=PLpiLOsNLsfmbeH-b865BwfH15W0sat02V&index=3\|800\|center}}

	Overall, the task of MT data pre-processing is to ''drop any distinctions that are not important for the output''.		Overall, the task of MT data pre-processing is to ''drop any distinctions that are not important for the output''.

Bojar: /* Script/Characters */ link to homoglyphs.net

2015-01-20T09:51:00Z

Script/Characters: link to homoglyphs.net

← Older revision		Revision as of 09:51, 20 January 2015
Line 48:		Line 48:
	=== Script/Characters ===		=== Script/Characters ===

	Unicode often provides many ways how to write a single character. For example, the letter "'''a'''" might be written with Latin or Cyrillic script.		Unicode often provides many ways how to write a single character. For example, the letter "'''a'''" might be written with Latin or Cyrillic script. A nice summary of Latin-like alphabets is available at [http://homoglyphs.net/ homoglyphs.net]

	Aside from seemingly identical, but differently encoded characters, problems commonly seen in data include:		Aside from seemingly identical, but differently encoded characters, problems commonly seen in data include:
Line 55:		Line 55:
	* Inconsistent letter case: English word '''I''' written in lowercase etc. (notorious e.g. in movie subtitles)		* Inconsistent letter case: English word '''I''' written in lowercase etc. (notorious e.g. in movie subtitles)
	* Various systematic mis-spellings -- all of these variants of "I'll" (I will) were observed in movie subtitles: '''i'll Ill l'll 1'll 1'11'''		* Various systematic mis-spellings -- all of these variants of "I'll" (I will) were observed in movie subtitles: '''i'll Ill l'll 1'll 1'11'''
	* Different symbols for various punctuation (quotes, dashes, apostrophe etc.)		* Different symbols for various punctuation (quotes, dashes, apostrophe etc.)

	=== VIM tips ===		=== VIM tips ===

Tamchyna at 12:28, 15 January 2015

2015-01-15T12:28:10Z

← Older revision		Revision as of 12:28, 15 January 2015
Line 104:		Line 104:
	== Exercises ==		== Exercises ==

	This is the first lecture accompanied by programming exercises. Before starting, you should follow the [[CodEx-Introduction instructions]] on how to use the CodEx submission system.		This is the first lecture accompanied by programming exercises. Before starting, you should follow the [[CodEx-Introduction\|instructions]] on how to use the CodEx submission system.

	Follow the links to see the description of each task and a submission interface with automatic evaluation of your solutions.		Follow the links to see the description of each task and a submission interface with automatic evaluation of your solutions.

Tamchyna at 12:27, 15 January 2015

2015-01-15T12:27:49Z

← Older revision		Revision as of 12:27, 15 January 2015
Line 101:		Line 101:

	It is essential for data tokenization to be '''consistent'''. All of our training data should conform to the same pre-processing scheme and an identical pipeline should be applied at test time (when our system runs and we translate new data).		It is essential for data tokenization to be '''consistent'''. All of our training data should conform to the same pre-processing scheme and an identical pipeline should be applied at test time (when our system runs and we translate new data).

			== Exercises ==

			This is the first lecture accompanied by programming exercises. Before starting, you should follow the [[CodEx-Introduction instructions]] on how to use the CodEx submission system.

			Follow the links to see the description of each task and a submission interface with automatic evaluation of your solutions.

			* [https://codex3.ms.mff.cuni.cz/codex-trans/?groupId=3&taskId=6&module=groups%2Ftasks&page=specification Lowercasing]
			* [https://codex3.ms.mff.cuni.cz/codex-trans/?groupId=3&taskId=7&module=groups%2Ftasks&page=specification Deaccenting]

Tamchyna at 12:24, 15 January 2015

2015-01-15T12:24:17Z

← Older revision		Revision as of 12:24, 15 January 2015
Line 5:		Line 5:
	\|data1 = [http://example.com web '''TODO'''] <br/> [http://www.youtube.com/watch?v=GDij7urWeOk Youtube]		\|data1 = [http://example.com web '''TODO'''] <br/> [http://www.youtube.com/watch?v=GDij7urWeOk Youtube]
	\|label2 = Exercises:		\|label2 = Exercises:
	\|data2 = [https://codex3.ms.mff.cuni.cz/codex-trans/?~~exerciseId~~=1&module=~~exercises~~&page=specification Lowercasing] <br/> [https://codex3.ms.mff.cuni.cz/codex-trans/?~~exerciseId~~=2&module=~~exercises~~&page=specification Deaccenting]		\|data2 = [https://codex3.ms.mff.cuni.cz/codex-trans/?groupId=3&taskId=6&module=groups%2Ftasks&page=specification Lowercasing] <br/> [https://codex3.ms.mff.cuni.cz/codex-trans/?groupId=3&taskId=7&module=groups%2Ftasks&page=specification Deaccenting]
	}}		}}

Bojar: 'MT' data preprocessing

2015-01-14T14:07:15Z

'MT' data preprocessing

← Older revision		Revision as of 14:07, 14 January 2015
Line 10:		Line 10:
	{{#ev:youtube\|GDij7urWeOk\|800\|center}}		{{#ev:youtube\|GDij7urWeOk\|800\|center}}

	Overall, the task of data pre-processing is to ''drop any distinctions that are not important for the output''.		Overall, the task of MT data pre-processing is to ''drop any distinctions that are not important for the output''.

	== Inspecting Text Data ==		== Inspecting Text Data ==

Tamchyna at 13:58, 13 January 2015

2015-01-13T13:58:54Z

← Older revision		Revision as of 13:58, 13 January 2015
Line 3:		Line 3:
	\|image = [[File:bear-with-us.png\|200px]]		\|image = [[File:bear-with-us.png\|200px]]
	\|label1 = Lecture video:		\|label1 = Lecture video:
	\|data1 = [http://example.com web '''TODO'''] <br/> [http://www.youtube.com/watch?v=~~ucSv4S4sCjs~~ Youtube]		\|data1 = [http://example.com web '''TODO'''] <br/> [http://www.youtube.com/watch?v=GDij7urWeOk Youtube]
	\|label2 = Exercises:		\|label2 = Exercises:
	\|data2 = [https://codex3.ms.mff.cuni.cz/codex-trans/?exerciseId=1&module=exercises&page=specification Lowercasing] <br/> [https://codex3.ms.mff.cuni.cz/codex-trans/?exerciseId=2&module=exercises&page=specification Deaccenting]		\|data2 = [https://codex3.ms.mff.cuni.cz/codex-trans/?exerciseId=1&module=exercises&page=specification Lowercasing] <br/> [https://codex3.ms.mff.cuni.cz/codex-trans/?exerciseId=2&module=exercises&page=specification Deaccenting]
	}}		}}

	{{#ev:youtube\|~~ucSv4S4sCjs~~\|800\|center}}		{{#ev:youtube\|GDij7urWeOk\|800\|center}}

	Overall, the task of data pre-processing is to ''drop any distinctions that are not important for the output''.		Overall, the task of data pre-processing is to ''drop any distinctions that are not important for the output''.

Tamchyna at 12:47, 13 January 2015

2015-01-13T12:47:27Z

@@ Line 73: / Line 73: @@
   ''':set bomb'''
   ''':set nobomb'''
 == Tokenization ==

Tamchyna at 12:36, 13 January 2015

2015-01-13T12:36:10Z

← Older revision		Revision as of 12:36, 13 January 2015
Line 4:		Line 4:
	\|label1 = Lecture video:		\|label1 = Lecture video:
	\|data1 = [http://example.com web '''TODO'''] <br/> [http://www.youtube.com/watch?v=ucSv4S4sCjs Youtube]		\|data1 = [http://example.com web '''TODO'''] <br/> [http://www.youtube.com/watch?v=ucSv4S4sCjs Youtube]
			\|label2 = Exercises:
			\|data2 = [https://codex3.ms.mff.cuni.cz/codex-trans/?exerciseId=1&module=exercises&page=specification Lowercasing] <br/> [https://codex3.ms.mff.cuni.cz/codex-trans/?exerciseId=2&module=exercises&page=specification Deaccenting]
	}}		}}