7 Commits

Author	SHA1	Message	Date
Arthur Zucker	a1790163fb	Changes for fast tokenizer (#5) ... - Add tokenizer.json (dcca07232bfb1028e499333730f868b87fd3d043) - Update unknown token (8852c40b30c9b7b981faf4fa77167fd862fd5fdb) - Move <|endoftext|> from added_tokens.json to vocab.json (3e9581879a6134abfb58f5788096027dd1756a63) Co-authored-by: Jonatan Kłosko <jonatanklosko@users.noreply.huggingface.co>	2023-01-25 10:17:42 +00:00
Arthur Zucker	27a139f678	Update `return_mask` param	2022-10-17 11:06:57 +00:00
Arthur Zucker	0996e56701	Upload processor	2022-10-04 14:58:01 +00:00
Arthur Zucker	c65924544a	Upload tokenizer	2022-09-28 08:15:02 +00:00
Arthur Zucker	8e72aa6038	Upload tokenizer	2022-09-27 09:55:50 +00:00
Arthur Zucker	1dff7eb67e	Upload processor	2022-09-27 07:04:56 +00:00
Arthur Zucker	25da2ff16c	Upload processor	2022-09-26 06:50:41 +00:00