跳转到正文
返回
技术笔记

Stanford CS336

学习过程、笔记、思考

课程介绍

CS336:从零开始构建属于自己的大模型。

本课程学习过程参考以下资料:

  1. B站课程链接:https://www.bilibili.com/video/BV1msTD6CE6j/?vd_source=66414b75f6db8197e95def95e244d230&p=1
  2. Github学习框架:https://github.com/bcefghj/learn-cs336/blob/main/docs/02-BPE%E5%88%86%E8%AF%8D%E5%99%A8%E5%8E%9F%E7%90%86%E4%B8%8E%E5%AE%9E%E7%8E%B0.md
  3. Stanford原始网站: https://cs336.stanford.edu/

Lesson 01

Overview & Tokenization

The Bitter Lesson

Wrong interpretation: scale is all that matters, algorithms don’t matter.

Right interpretation: algorithms that scale is what matters.

accracy=efficiency * resources

Tokenization

Tokenizer

把string转化为tokens序列 下面就是一个string2tokens的例子,其中tokens包含的是每个token的indices。

string = "Hello, 🌍! 你好!"
tokens = [15496, 11, 995, 0]

与tokens对应的是Vocabulary(词表),代表了所有有可能出现的tokens,indices就是token在Vocabulary里面的序号。

string->tokens称为encode,反之tokens->string则称为decode

从string到tokens,整个序列的元素数量减少了,我们把string的字节数与tokens数的比例称为Compression_ratio,用来衡量Tokenizer的性能

Three types of tokenizer

Character-based

Each Unicode Character can be converted into a code point (integer) via ord, and it can be converted back via chr

assert ord("a") == 97
assert ord("🌍") == 127757
assert chr(97) == "a"
assert chr(127757) == "🌍"

But there are approximately 150k unicode characters,

Vocabulary_size = max(indices) + 1

Bytes-based

Unicode strings can be represented as a sequence of bytes, which can be represented by integers between 0 and 255.

assert bytes("a", encoding="utf-8") == b"a"
assert bytes("🌍", encoding="utf-8") == b"\xf0\x9f\x8c\x8d"

Vocabulary_size = 256, but Compression_ratio = 1, which means the sequences will be too long The context of transformer is limited, so it is not good.

Words-based

First, split strings to words. Regular Expression keeps all alphanumeric characters together.

string = "I'll say supercalifragilisticexpialidocious!"
segments = regex.findall(r"\w+|.", string)  # @inspect segments

Then turn it into tokenizer. We can build a mapping from each segment into an integer.

Problems:

New words we haven’t seen during training get a special UNK token, which is ugly and can mess up perplexity calculations.

BPE

Basic idea: train the tokenizer on raw text to automatically determine the vocabulary.

让Tokenizer自动决定如何构建vocabulary

Intuition: common sequences of characters are represented by a single token, rare sequences are represented by many tokens.

首先,我们计算每对相邻的token对(初始每个token都是一个字节)的出现次数,然后把出现次数最多的token对合并,作为vocab[++max_indices]的token。再重新统计,合并下一个token对,知道合并次数满足要求或者到达Vocabulary容量上限为止。



下一篇
从零开始,搭建一个属于自己的博客