pg_tiktoken

在PostgreSQL中计算OpenAI使用的Token数

概览

扩展包名	版本	分类	许可证	语言
`pg_tiktoken`	`0.0.1`	RAG	Apache-2.0	Rust

ID	扩展名	Bin	Lib	Load	Create	Trust	Reloc	模式
1870	`pg_tiktoken`	否	是	否	是	否	否	-

相关扩展	`pg_tiktoken_c` `pg_tokenizer` `pg_jieba` `pg_cjk_parser` `zhparser` `pg_bigm` `pgroonga` `dict_xsyn`

版本

类型	仓库	版本	PG 大版本	包名	依赖
EXT	PIGSTY	`0.0.1`	1817161514	`pg_tiktoken`	-
RPM	PIGSTY	`0.0.1`	1817161514	`pg_tiktoken_$v`	-
DEB	PIGSTY	`0.0.1`	1817161514	`postgresql-$v-pg-tiktoken`	-

OS / PG	PG18	PG17	PG16	PG15	PG14
el8.x86_64	PIGSTY 0.0.1 el8.x86_64.pg18 : pg_tiktoken_18 pg_tiktoken_18-0.0.1-4PIGSTY.el8.x86_64.rpm PIGSTY · 0.0.1 · 2.3MiB	PIGSTY 0.0.1 el8.x86_64.pg17 : pg_tiktoken_17 pg_tiktoken_17-0.0.1-4PIGSTY.el8.x86_64.rpm PIGSTY · 0.0.1 · 2.3MiB	PIGSTY 0.0.1 el8.x86_64.pg16 : pg_tiktoken_16 pg_tiktoken_16-0.0.1-4PIGSTY.el8.x86_64.rpm PIGSTY · 0.0.1 · 2.3MiB	PIGSTY 0.0.1 el8.x86_64.pg15 : pg_tiktoken_15 pg_tiktoken_15-0.0.1-4PIGSTY.el8.x86_64.rpm PIGSTY · 0.0.1 · 2.3MiB	PIGSTY 0.0.1 el8.x86_64.pg14 : pg_tiktoken_14 pg_tiktoken_14-0.0.1-4PIGSTY.el8.x86_64.rpm PIGSTY · 0.0.1 · 2.3MiB
el8.aarch64	PIGSTY 0.0.1 el8.aarch64.pg18 : pg_tiktoken_18 pg_tiktoken_18-0.0.1-4PIGSTY.el8.aarch64.rpm PIGSTY · 0.0.1 · 2.2MiB	PIGSTY 0.0.1 el8.aarch64.pg17 : pg_tiktoken_17 pg_tiktoken_17-0.0.1-4PIGSTY.el8.aarch64.rpm PIGSTY · 0.0.1 · 2.2MiB	PIGSTY 0.0.1 el8.aarch64.pg16 : pg_tiktoken_16 pg_tiktoken_16-0.0.1-4PIGSTY.el8.aarch64.rpm PIGSTY · 0.0.1 · 2.2MiB	PIGSTY 0.0.1 el8.aarch64.pg15 : pg_tiktoken_15 pg_tiktoken_15-0.0.1-4PIGSTY.el8.aarch64.rpm PIGSTY · 0.0.1 · 2.2MiB	PIGSTY 0.0.1 el8.aarch64.pg14 : pg_tiktoken_14 pg_tiktoken_14-0.0.1-4PIGSTY.el8.aarch64.rpm PIGSTY · 0.0.1 · 2.2MiB
el9.x86_64	PIGSTY 0.0.1 el9.x86_64.pg18 : pg_tiktoken_18 pg_tiktoken_18-0.0.1-4PIGSTY.el9.x86_64.rpm PIGSTY · 0.0.1 · 2.2MiB	PIGSTY 0.0.1 el9.x86_64.pg17 : pg_tiktoken_17 pg_tiktoken_17-0.0.1-4PIGSTY.el9.x86_64.rpm PIGSTY · 0.0.1 · 2.2MiB	PIGSTY 0.0.1 el9.x86_64.pg16 : pg_tiktoken_16 pg_tiktoken_16-0.0.1-4PIGSTY.el9.x86_64.rpm PIGSTY · 0.0.1 · 2.2MiB	PIGSTY 0.0.1 el9.x86_64.pg15 : pg_tiktoken_15 pg_tiktoken_15-0.0.1-4PIGSTY.el9.x86_64.rpm PIGSTY · 0.0.1 · 2.2MiB	PIGSTY 0.0.1 el9.x86_64.pg14 : pg_tiktoken_14 pg_tiktoken_14-0.0.1-4PIGSTY.el9.x86_64.rpm PIGSTY · 0.0.1 · 2.2MiB
el9.aarch64	PIGSTY 0.0.1 el9.aarch64.pg18 : pg_tiktoken_18 pg_tiktoken_18-0.0.1-4PIGSTY.el9.aarch64.rpm PIGSTY · 0.0.1 · 2.2MiB	PIGSTY 0.0.1 el9.aarch64.pg17 : pg_tiktoken_17 pg_tiktoken_17-0.0.1-4PIGSTY.el9.aarch64.rpm PIGSTY · 0.0.1 · 2.2MiB	PIGSTY 0.0.1 el9.aarch64.pg16 : pg_tiktoken_16 pg_tiktoken_16-0.0.1-4PIGSTY.el9.aarch64.rpm PIGSTY · 0.0.1 · 2.2MiB	PIGSTY 0.0.1 el9.aarch64.pg15 : pg_tiktoken_15 pg_tiktoken_15-0.0.1-4PIGSTY.el9.aarch64.rpm PIGSTY · 0.0.1 · 2.2MiB	PIGSTY 0.0.1 el9.aarch64.pg14 : pg_tiktoken_14 pg_tiktoken_14-0.0.1-4PIGSTY.el9.aarch64.rpm PIGSTY · 0.0.1 · 2.2MiB
el10.x86_64	PIGSTY 0.0.1 el10.x86_64.pg18 : pg_tiktoken_18 pg_tiktoken_18-0.0.1-4PIGSTY.el10.x86_64.rpm PIGSTY · 0.0.1 · 2.2MiB	PIGSTY 0.0.1 el10.x86_64.pg17 : pg_tiktoken_17 pg_tiktoken_17-0.0.1-4PIGSTY.el10.x86_64.rpm PIGSTY · 0.0.1 · 2.2MiB	PIGSTY 0.0.1 el10.x86_64.pg16 : pg_tiktoken_16 pg_tiktoken_16-0.0.1-4PIGSTY.el10.x86_64.rpm PIGSTY · 0.0.1 · 2.2MiB	PIGSTY 0.0.1 el10.x86_64.pg15 : pg_tiktoken_15 pg_tiktoken_15-0.0.1-4PIGSTY.el10.x86_64.rpm PIGSTY · 0.0.1 · 2.2MiB	PIGSTY 0.0.1 el10.x86_64.pg14 : pg_tiktoken_14 pg_tiktoken_14-0.0.1-4PIGSTY.el10.x86_64.rpm PIGSTY · 0.0.1 · 2.2MiB
el10.aarch64	PIGSTY 0.0.1 el10.aarch64.pg18 : pg_tiktoken_18 pg_tiktoken_18-0.0.1-4PIGSTY.el10.aarch64.rpm PIGSTY · 0.0.1 · 2.2MiB	PIGSTY 0.0.1 el10.aarch64.pg17 : pg_tiktoken_17 pg_tiktoken_17-0.0.1-4PIGSTY.el10.aarch64.rpm PIGSTY · 0.0.1 · 2.2MiB	PIGSTY 0.0.1 el10.aarch64.pg16 : pg_tiktoken_16 pg_tiktoken_16-0.0.1-4PIGSTY.el10.aarch64.rpm PIGSTY · 0.0.1 · 2.2MiB	PIGSTY 0.0.1 el10.aarch64.pg15 : pg_tiktoken_15 pg_tiktoken_15-0.0.1-4PIGSTY.el10.aarch64.rpm PIGSTY · 0.0.1 · 2.2MiB	PIGSTY 0.0.1 el10.aarch64.pg14 : pg_tiktoken_14 pg_tiktoken_14-0.0.1-4PIGSTY.el10.aarch64.rpm PIGSTY · 0.0.1 · 2.2MiB
d12.x86_64	PIGSTY 0.0.1 d12.x86_64.pg18 : postgresql-18-pg-tiktoken postgresql-18-pg-tiktoken_0.0.1-3PIGSTY~bookworm_amd64.deb PIGSTY · 0.0.1 · 1.9MiB	PIGSTY 0.0.1 d12.x86_64.pg17 : postgresql-17-pg-tiktoken postgresql-17-pg-tiktoken_0.0.1-3PIGSTY~bookworm_amd64.deb PIGSTY · 0.0.1 · 1.9MiB	PIGSTY 0.0.1 d12.x86_64.pg16 : postgresql-16-pg-tiktoken postgresql-16-pg-tiktoken_0.0.1-3PIGSTY~bookworm_amd64.deb PIGSTY · 0.0.1 · 1.9MiB	PIGSTY 0.0.1 d12.x86_64.pg15 : postgresql-15-pg-tiktoken postgresql-15-pg-tiktoken_0.0.1-3PIGSTY~bookworm_amd64.deb PIGSTY · 0.0.1 · 1.8MiB	PIGSTY 0.0.1 d12.x86_64.pg14 : postgresql-14-pg-tiktoken postgresql-14-pg-tiktoken_0.0.1-3PIGSTY~bookworm_amd64.deb PIGSTY · 0.0.1 · 1.8MiB
d12.aarch64	PIGSTY 0.0.1 d12.aarch64.pg18 : postgresql-18-pg-tiktoken postgresql-18-pg-tiktoken_0.0.1-3PIGSTY~bookworm_arm64.deb PIGSTY · 0.0.1 · 1.7MiB	PIGSTY 0.0.1 d12.aarch64.pg17 : postgresql-17-pg-tiktoken postgresql-17-pg-tiktoken_0.0.1-3PIGSTY~bookworm_arm64.deb PIGSTY · 0.0.1 · 1.7MiB	PIGSTY 0.0.1 d12.aarch64.pg16 : postgresql-16-pg-tiktoken postgresql-16-pg-tiktoken_0.0.1-3PIGSTY~bookworm_arm64.deb PIGSTY · 0.0.1 · 1.7MiB	PIGSTY 0.0.1 d12.aarch64.pg15 : postgresql-15-pg-tiktoken postgresql-15-pg-tiktoken_0.0.1-3PIGSTY~bookworm_arm64.deb PIGSTY · 0.0.1 · 1.7MiB	PIGSTY 0.0.1 d12.aarch64.pg14 : postgresql-14-pg-tiktoken postgresql-14-pg-tiktoken_0.0.1-3PIGSTY~bookworm_arm64.deb PIGSTY · 0.0.1 · 1.7MiB
d13.x86_64	PIGSTY 0.0.1 d13.x86_64.pg18 : postgresql-18-pg-tiktoken postgresql-18-pg-tiktoken_0.0.1-3PIGSTY~trixie_amd64.deb PIGSTY · 0.0.1 · 1.9MiB	PIGSTY 0.0.1 d13.x86_64.pg17 : postgresql-17-pg-tiktoken postgresql-17-pg-tiktoken_0.0.1-3PIGSTY~trixie_amd64.deb PIGSTY · 0.0.1 · 1.9MiB	PIGSTY 0.0.1 d13.x86_64.pg16 : postgresql-16-pg-tiktoken postgresql-16-pg-tiktoken_0.0.1-3PIGSTY~trixie_amd64.deb PIGSTY · 0.0.1 · 1.9MiB	PIGSTY 0.0.1 d13.x86_64.pg15 : postgresql-15-pg-tiktoken postgresql-15-pg-tiktoken_0.0.1-3PIGSTY~trixie_amd64.deb PIGSTY · 0.0.1 · 1.8MiB	PIGSTY 0.0.1 d13.x86_64.pg14 : postgresql-14-pg-tiktoken postgresql-14-pg-tiktoken_0.0.1-3PIGSTY~trixie_amd64.deb PIGSTY · 0.0.1 · 1.8MiB
d13.aarch64	PIGSTY 0.0.1 d13.aarch64.pg18 : postgresql-18-pg-tiktoken postgresql-18-pg-tiktoken_0.0.1-3PIGSTY~trixie_arm64.deb PIGSTY · 0.0.1 · 1.7MiB	PIGSTY 0.0.1 d13.aarch64.pg17 : postgresql-17-pg-tiktoken postgresql-17-pg-tiktoken_0.0.1-3PIGSTY~trixie_arm64.deb PIGSTY · 0.0.1 · 1.7MiB	PIGSTY 0.0.1 d13.aarch64.pg16 : postgresql-16-pg-tiktoken postgresql-16-pg-tiktoken_0.0.1-3PIGSTY~trixie_arm64.deb PIGSTY · 0.0.1 · 1.7MiB	PIGSTY 0.0.1 d13.aarch64.pg15 : postgresql-15-pg-tiktoken postgresql-15-pg-tiktoken_0.0.1-3PIGSTY~trixie_arm64.deb PIGSTY · 0.0.1 · 1.7MiB	PIGSTY 0.0.1 d13.aarch64.pg14 : postgresql-14-pg-tiktoken postgresql-14-pg-tiktoken_0.0.1-3PIGSTY~trixie_arm64.deb PIGSTY · 0.0.1 · 1.7MiB
u22.x86_64	PIGSTY 0.0.1 u22.x86_64.pg18 : postgresql-18-pg-tiktoken postgresql-18-pg-tiktoken_0.0.1-3PIGSTY~jammy_amd64.deb PIGSTY · 0.0.1 · 2.1MiB	PIGSTY 0.0.1 u22.x86_64.pg17 : postgresql-17-pg-tiktoken postgresql-17-pg-tiktoken_0.0.1-3PIGSTY~jammy_amd64.deb PIGSTY · 0.0.1 · 2.1MiB	PIGSTY 0.0.1 u22.x86_64.pg16 : postgresql-16-pg-tiktoken postgresql-16-pg-tiktoken_0.0.1-3PIGSTY~jammy_amd64.deb PIGSTY · 0.0.1 · 2.1MiB	PIGSTY 0.0.1 u22.x86_64.pg15 : postgresql-15-pg-tiktoken postgresql-15-pg-tiktoken_0.0.1-3PIGSTY~jammy_amd64.deb PIGSTY · 0.0.1 · 2.1MiB	PIGSTY 0.0.1 u22.x86_64.pg14 : postgresql-14-pg-tiktoken postgresql-14-pg-tiktoken_0.0.1-3PIGSTY~jammy_amd64.deb PIGSTY · 0.0.1 · 2.1MiB
u22.aarch64	PIGSTY 0.0.1 u22.aarch64.pg18 : postgresql-18-pg-tiktoken postgresql-18-pg-tiktoken_0.0.1-3PIGSTY~jammy_arm64.deb PIGSTY · 0.0.1 · 2.0MiB	PIGSTY 0.0.1 u22.aarch64.pg17 : postgresql-17-pg-tiktoken postgresql-17-pg-tiktoken_0.0.1-3PIGSTY~jammy_arm64.deb PIGSTY · 0.0.1 · 2.0MiB	PIGSTY 0.0.1 u22.aarch64.pg16 : postgresql-16-pg-tiktoken postgresql-16-pg-tiktoken_0.0.1-3PIGSTY~jammy_arm64.deb PIGSTY · 0.0.1 · 2.0MiB	PIGSTY 0.0.1 u22.aarch64.pg15 : postgresql-15-pg-tiktoken postgresql-15-pg-tiktoken_0.0.1-3PIGSTY~jammy_arm64.deb PIGSTY · 0.0.1 · 2.0MiB	PIGSTY 0.0.1 u22.aarch64.pg14 : postgresql-14-pg-tiktoken postgresql-14-pg-tiktoken_0.0.1-3PIGSTY~jammy_arm64.deb PIGSTY · 0.0.1 · 2.0MiB
u24.x86_64	PIGSTY 0.0.1 u24.x86_64.pg18 : postgresql-18-pg-tiktoken postgresql-18-pg-tiktoken_0.0.1-3PIGSTY~noble_amd64.deb PIGSTY · 0.0.1 · 2.1MiB	PIGSTY 0.0.1 u24.x86_64.pg17 : postgresql-17-pg-tiktoken postgresql-17-pg-tiktoken_0.0.1-3PIGSTY~noble_amd64.deb PIGSTY · 0.0.1 · 2.1MiB	PIGSTY 0.0.1 u24.x86_64.pg16 : postgresql-16-pg-tiktoken postgresql-16-pg-tiktoken_0.0.1-3PIGSTY~noble_amd64.deb PIGSTY · 0.0.1 · 2.1MiB	PIGSTY 0.0.1 u24.x86_64.pg15 : postgresql-15-pg-tiktoken postgresql-15-pg-tiktoken_0.0.1-3PIGSTY~noble_amd64.deb PIGSTY · 0.0.1 · 2.1MiB	PIGSTY 0.0.1 u24.x86_64.pg14 : postgresql-14-pg-tiktoken postgresql-14-pg-tiktoken_0.0.1-3PIGSTY~noble_amd64.deb PIGSTY · 0.0.1 · 2.1MiB
u24.aarch64	PIGSTY 0.0.1 u24.aarch64.pg18 : postgresql-18-pg-tiktoken postgresql-18-pg-tiktoken_0.0.1-3PIGSTY~noble_arm64.deb PIGSTY · 0.0.1 · 2.0MiB	PIGSTY 0.0.1 u24.aarch64.pg17 : postgresql-17-pg-tiktoken postgresql-17-pg-tiktoken_0.0.1-3PIGSTY~noble_arm64.deb PIGSTY · 0.0.1 · 2.0MiB	PIGSTY 0.0.1 u24.aarch64.pg16 : postgresql-16-pg-tiktoken postgresql-16-pg-tiktoken_0.0.1-3PIGSTY~noble_arm64.deb PIGSTY · 0.0.1 · 2.0MiB	PIGSTY 0.0.1 u24.aarch64.pg15 : postgresql-15-pg-tiktoken postgresql-15-pg-tiktoken_0.0.1-3PIGSTY~noble_arm64.deb PIGSTY · 0.0.1 · 2.0MiB	PIGSTY 0.0.1 u24.aarch64.pg14 : postgresql-14-pg-tiktoken postgresql-14-pg-tiktoken_0.0.1-3PIGSTY~noble_arm64.deb PIGSTY · 0.0.1 · 2.0MiB
u26.x86_64	PIGSTY 0.0.1 u26.x86_64.pg18 : postgresql-18-pg-tiktoken postgresql-18-pg-tiktoken_0.0.1-3PIGSTY~resolute_amd64.deb PIGSTY · 0.0.1 · 2.1MiB	PIGSTY 0.0.1 u26.x86_64.pg17 : postgresql-17-pg-tiktoken postgresql-17-pg-tiktoken_0.0.1-3PIGSTY~resolute_amd64.deb PIGSTY · 0.0.1 · 2.1MiB	PIGSTY 0.0.1 u26.x86_64.pg16 : postgresql-16-pg-tiktoken postgresql-16-pg-tiktoken_0.0.1-3PIGSTY~resolute_amd64.deb PIGSTY · 0.0.1 · 2.1MiB	PIGSTY 0.0.1 u26.x86_64.pg15 : postgresql-15-pg-tiktoken postgresql-15-pg-tiktoken_0.0.1-3PIGSTY~resolute_amd64.deb PIGSTY · 0.0.1 · 2.1MiB	PIGSTY 0.0.1 u26.x86_64.pg14 : postgresql-14-pg-tiktoken postgresql-14-pg-tiktoken_0.0.1-3PIGSTY~resolute_amd64.deb PIGSTY · 0.0.1 · 2.1MiB
u26.aarch64	PIGSTY 0.0.1 u26.aarch64.pg18 : postgresql-18-pg-tiktoken postgresql-18-pg-tiktoken_0.0.1-3PIGSTY~resolute_arm64.deb PIGSTY · 0.0.1 · 2.0MiB	PIGSTY 0.0.1 u26.aarch64.pg17 : postgresql-17-pg-tiktoken postgresql-17-pg-tiktoken_0.0.1-3PIGSTY~resolute_arm64.deb PIGSTY · 0.0.1 · 2.0MiB	PIGSTY 0.0.1 u26.aarch64.pg16 : postgresql-16-pg-tiktoken postgresql-16-pg-tiktoken_0.0.1-3PIGSTY~resolute_arm64.deb PIGSTY · 0.0.1 · 2.0MiB	PIGSTY 0.0.1 u26.aarch64.pg15 : postgresql-15-pg-tiktoken postgresql-15-pg-tiktoken_0.0.1-3PIGSTY~resolute_arm64.deb PIGSTY · 0.0.1 · 2.0MiB	PIGSTY 0.0.1 u26.aarch64.pg14 : postgresql-14-pg-tiktoken postgresql-14-pg-tiktoken_0.0.1-3PIGSTY~resolute_arm64.deb PIGSTY · 0.0.1 · 2.0MiB

构建

您可以使用 pig build 命令构建 pg_tiktoken 扩展的 RPM / DEB 包：

BASH

pig build pkg pg_tiktoken         # 构建 RPM / DEB 包

安装

您可以直接安装 pg_tiktoken 扩展包的预置二进制包，首先确保 PGDG 和 PIGSTY 仓库已经添加并启用：

BASH

pig repo add pgsql -u          # 添加仓库并更新缓存

使用 pig 或者是 apt/yum/dnf 安装扩展：

安装

BASH

pig install pg_tiktoken;          # 当前活跃 PG 版本安装

pig

BASH

pig ext install -y pg_tiktoken -v 18  # PG 18
pig ext install -y pg_tiktoken -v 17  # PG 17
pig ext install -y pg_tiktoken -v 16  # PG 16
pig ext install -y pg_tiktoken -v 15  # PG 15
pig ext install -y pg_tiktoken -v 14  # PG 14

dnf

BASH

dnf install -y pg_tiktoken_18       # PG 18
dnf install -y pg_tiktoken_17       # PG 17
dnf install -y pg_tiktoken_16       # PG 16
dnf install -y pg_tiktoken_15       # PG 15
dnf install -y pg_tiktoken_14       # PG 14

apt

BASH

apt install -y postgresql-18-pg-tiktoken   # PG 18
apt install -y postgresql-17-pg-tiktoken   # PG 17
apt install -y postgresql-16-pg-tiktoken   # PG 16
apt install -y postgresql-15-pg-tiktoken   # PG 15
apt install -y postgresql-14-pg-tiktoken   # PG 14

创建扩展：

SQL

CREATE EXTENSION pg_tiktoken;

用法

pg_tiktoken：用于 PostgreSQL 中 OpenAI 模型的 tiktoken 分词器。来源：README.md

pg_tiktoken 是一个 PostgreSQL 扩展，使用 OpenAI 的 tiktoken 库提供输入分词功能。它允许你直接在 SQL 中计数和编码 token，这在处理 OpenAI 模型的输入长度限制时非常有用。

函数

tiktoken_count

计算给定编码或模型的 token 数量：

SQL

SELECT tiktoken_count('p50k_edit', 'A long time ago in a galaxy far, far away');
 tiktoken_count
----------------
             11
(1 row)

tiktoken_encode

获取给定编码或模型的 token ID：

SQL

SELECT tiktoken_encode('cl100k_base', 'A long time ago in a galaxy far, far away');
                  tiktoken_encode
----------------------------------------------------
 {32,1317,892,4227,304,264,34261,3117,11,3117,3201}
(1 row)

tiktoken_count 和 tiktoken_encode 的第一个参数都可以接受编码名称或 OpenAI 模型名称。

支持的模型

编码名称	OpenAI 模型
`cl100k_base`	ChatGPT 模型, `text-embedding-ada-002`
`p50k_base`	代码模型, `text-davinci-002`, `text-davinci-003`
`p50k_edit`	编辑模型如 `text-davinci-edit-001`, `code-davinci-edit-001`
`r50k_base`（或 `gpt2`）	GPT-3 模型如 `davinci`

意见反馈

这个页面对您有帮助吗？

感谢反馈！请告诉我们如何改进。

抱歉给您带来不便。请告诉我们如何改进。