使用erlang做後端、flutter做前端開發的一款開源的即時聊天解決方案（基於erlang/otp的高性能web框架cowboy 做後端服務，用”8核16G 主機（100萬PPS）”壓測，保持100萬+TCP穩定在線90分鐘以上）支援基於webrtc 一對一視頻通話功能

Jiking 2023-05-10 Article

基於cowboy(Small, fast, modern HTTP server for Erlang/OTP) 的即時聊天后端服務，使用”阿里雲8核16G ecs.sn1ne.2xlarge主機（100萬PPS）”壓測，保持100萬+TCP，90分鐘以上，詳細測試件測試文檔

因為我是中國人，所以選擇了木蘭寬鬆許可證, 第2版

https://ninenines.eu/docs/en/cowboy/2.9/guide/getting_started/

Version 0.1.0

力求基於“語義化版本控制的規範”(語義化版本2.0.0)實施版本管理.

Strive to implement version management based on “Specification for Semantic version Control”(Semantic Versioning 2.0.0).

環境依賴(Environment depends on)

數據結構(./doc/postgresql/vsn0.1)開發中有變動，以第一個發布版為準；目前改成基於PostgreSQL15 開發

There are changes in the data structure (./doc/postgresql/vsn0.1) under development. It is currently based on PostgreSQL15

Erlang/OTP 23 / Erlang/OTP 24 / Erlang/OTP 25

數據庫PostgreSQL15

kerl

// 列表可安装的版本号
kerl list releases

kerl build 24.3.4.2 / kerl delete build 24.3.3


kerl list builds

kerl install 24.3.4.2 ~/kerl/24.3.4.2

. /Users/leeyi/kerl/24.3.4.2/activate

Later on, you can leave the installation typing:
kerl_deactivate

Anytime you can check which installation, if any, is currently active with:
kerl active

Redis Stack

查看

Grafana

Grafana 是一個知名且廣泛使用的開源應用程式監控工具。

介紹Grafana 的Redis 數據源插件


make new t=cowboy.http n=handler_passport
make new t=cowboy.ws n=handler_websocket

make new t=cowboy.middleware n=middleware_demo
make new t=cowboy.middleware n=middleware_auth
make new t=gen_server n=server_account

make distclean

// 我添加的模板 以imboy开头
make new t=imboy.rest_handler n=demo_handler
make new t=imboy.logic n=demo_logic
make new t=imboy.repository n=demo_repo

make new t=imboy.rest_handler n=demo2_handler && make new t=imboy.logic n=demo2_logic && make new t=imboy.repository n=demo2_repo

make list-templates
Available templates:
    cowboy.http
    cowboy.loop
    cowboy.rest
    cowboy.ws
    cowboy_http
    cowboy_loop
    cowboy_rest
    cowboy_ws
    gen_fsm
    gen_server
    gen_statem
    imboy.logic
    imboy.repository
    imboy.rest_handler
    module
    ranch_protocol
    supervisor

make run

// on Mac
IMBOYENV=prod make run
IMBOYENV=test make run
IMBOYENV=dev make run
IMBOYENV=local make run

// on CentOS8 OR macOS
export IMBOYENV='local' && make run

observer_cli:start().

make new t=gen_server n=server_demo

make dialyze

make

make rel

make help
  rel           Build a release for this project, if applicable

在另一個shelll裡面執行

erl> help().
    lm()       -- load all modified modules

// 更新 erlang.mk
make erlang-mk

edoc

link http://erlang.org/doc/apps/edoc/chapter.html#Introduction

test

分析工具(Analysis tool)

make dialyze

代码格式工具
get from https://github.com/sile/efmt/releases

VERSION=0.14.1
curl -L https://github.com/sile/efmt/releases/download/${VERSION}/efmt-${VERSION}.x86_64-unknown-linux-musl -o efmt
chmod +x efmt
./efmt

./efmt -c src/websocket_logic.erl
./efmt -w src/websocket_logic.erl

IMBOYENV=prod make rel
IMBOYENV=test make rel
IMBOYENV=dev make rel -j8
IMBOYENV=local make rel

複製程式碼到特定的目錄(Copy code to a specific directory)

cp ./_rel/imboy/imboy-1.0.0.tar.gz
// or
scp ./_rel/imboy/imboy-1.0.0.tar.gz root@192.168.2.207:/usr/local/imboy/

去啟動服務(To start the service)


mkdir -p /usr/local/imboy

cp ./_rel/imboy/imboy-1.0.0.tar.gz /usr/local/imboy/

cd /usr/local/imboy

tar -xzf imboy-1.0.0.tar.gz

bin/imboy console

bin/imboy start

bin/imboy restart

bin/imboy stop

更新發布(updates)

link https://erlang.mk/guide/relx.html

For the purpose of this section, assume the initial release version was 1, and the new version is 2. The name of the release will be example.

Once all this is done, you can build the tarball for the release upgrade:

This will create an archive at the root directory of the release, $RELX_OUTPUT_DIR/example/example-2.tar.gz.

Move the archive to the correct location on the running node. From the release’s root directory:

$ mkdir releases/2/
$ mv path/to/example-2.tar.gz releases/2/

Finally, upgrade the release:

$ bin/example_release upgrade "2/example_release"

scp ./_rel/imboy/imboy-0.1.1.tar.gz root@120.24.63.33:/usr/local/imboy

mv imboy-0.1.1.tar.gz releases/0.1.1/
bin/imboy upgrade "0.1.1/imboy"

bin/imboy downgrade "0.1.0/imboy"

Your release was upgraded!

imboy.appup

{"0.2.0",
    所有版本"0.1.*"升级到版本"0.2.0",重启应用
   [{"0.1\\.[0-9]+", [{restart_application, imboy_app}
             ]}],
    版本"0.2.0"降级到所有版本"0.1.*",重启应用
   [{"0.1\\.[0-9]+", [{restart_application, imboy_app}
             ]}]
}.

+K true
开启epoll调度，在linux中开启epoll，会大大增加调度的效率

+A 1024
异步线程池，为某些port调用服务

+P 2048000
最大进程数

+Q 2048000
最大port数

+sbt db
绑定调度器，绑定后调度器的任务队列不会在各个CPU线程之间跃迁，结合sub使用，可以让CPU负载均衡的同时也避免了大量的跃迁发生。

注意：一个linux系统中，最好只有一个evm开启此选项，若同时有多个erlang虚拟机在系统中运行，还是关闭为好


+sub true
开启CPU负载均衡，false的时候是采用的CPU密集调度策略，优先在某个CPU线程上运行任务，直到该CPU负载较高为止。

+swct eager
此选项设置为eager后，CPU将更频繁的被唤醒，可以增加CPU利用率

+spp true
开启并行port并行调度队列，当开启后会大大增加系统吞吐量，如果关闭，则会牺牲吞吐量换取更低的延迟。

+zdbbl 65536
分布式erlang的端口buffer大小，当buffer满的时候，向分布式的远程端口发送消息会阻塞


打开档案数 for mac
sudo launchctl limit maxfiles
sudo launchctl limit maxfiles 2097152 2097152
sudo ulimit -n 2097152

sysctl net.inet.ip.portrange.first net.inet.ip.portrange.last

## 及高范围
net.inet.ip.portrange.hifirst: 49152
net.inet.ip.portrange.hilast: 65535

sysctl -w net.inet.ip.portrange.first=1025
sysctl -w net.inet.ip.portrange.last=655350
sysctl -w net.inet.ip.tcp_rmem=655350


HAProxy + Docker * N + K8S + mnesia 集群
erlang:system_info(port_limit).

locust -f src/imboy.py --no-web -c 20000 -r 1000 -t 600s --logfile=logs/imboy-no-web.log
length(chat_store_repo:lookall()).

参考：

http://m.udpwork.com/item/11782.html
https://cloud.tencent.com/developer/article/1422476
https://www.yuanmomo.net/2019/07/26/mac-max-connections-config/
https://colobu.com/2014/09/18/linux-tcpip-tuning/

https://www.cnblogs.com/duanxz/p/4464178.html 单伺服器最大tcp连接数及调优汇总

https://blog.51cto.com/yaocoder/1312821

http://hk.uwenku.com/question/p-tgiqupmb-oc.html

https://knowledge.zhaoweiguo.com/8tools/mqtts/emqtts/emqtt_tune.html

https://studygolang.com/articles/2416

https://www.iteye.com/blog/mryufeng-475003  erlang 节点间通讯的通道微调

http://www.wangxingrong.com.cn/archives/tag/百万并发连接伺服器

https://qunfei.wordpress.com/2016/09/20/from-c10k-to-c100k-problem-push-over-1000000-messages-to-web-clients-on-1-machine-simultaneously/

https://stackoverflow.com/questions/32711242/erlang-simultaneously-connect-1m-clients

https://colobu.com/2015/05/22/implement-C1000K-servers-by-spray-netty-undertow-and-node-js

https://blog.csdn.net/zcc_0015/article/details/26407683 Linux下基于Erlang的高并发TCP连接压力实验

https://github.com/smallnest/C1000K-Servers

100万并发连接伺服器笔记之Erlang完成1M并发连接目标 https://blog.csdn.net/shallowgrave/article/details/19990345?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromBaidu-5.nonecase&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromBaidu-5.nonecase

docker run -it –rm –name imboy-1 -p 9801:9800 -v “$PWD”:/usr/src/imboy -w /usr/src/imboy erlang

// 後台運行docker-compose up -d docker-compose -f docker-local.yml up -d

下面的命令增加了19個IP地址，其中一個給服務器用sudo ifconfig lo0 alias 127.0.0.10 sudo ifconfig lo0 alias 127.0.0.11 length(chat_store_repo:lookall()).

sudo ifconfig lo0 -alias 127.0.0.10 sudo ifconfig lo0 -alias 127.0.0.11

Erlang虛擬機默認的端口上限為65536, erlang17通過erl +Q 1000000可以修改端口上限為1000000,利用erlang:system_info(port_limit)進行查詢，系統可以打開的最大档案描述符可以通過erlang:system_info(check_io)中的max_fds進行查看，查看系統當前port數量可以用erlang:length(erlang:ports())得到

erlang:system_info(port_limit) erlang:system_info(check_io) erlang:length(erlang:ports()).

Pid = spawn(fun() -> etop:start([{output, text}, {interval, 1}, {lines, 20}, {sort, memory}]) end).

查看TCP 数量
netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
ESTABLISHED 28705

free -h
              total        used        free      shared  buff/cache   available
Mem:           3.7G        2.8G        774M        452K        140M        726M
Swap:            0B          0B          0B

查看 pid
 pmap -d 6380

erl -name ws2@127.0.0.1 -setcookie imboy -hidden net_adm:ping(‘imboy@127.0.0.1’). Ctrl + G h r ‘imboy@127.0.0.1’ j c 2

cowboy_websocket 異步消息websocket close 傳遞參數

如何動態加載配置档案

ifeq ($(ENV),prod)
    RELX_CONFIG = $(CURDIR)/relx.prod.config
else ifeq ($(ENV),test)
    RELX_CONFIG = $(CURDIR)/relx.test.config
else ifeq ($(ENV),dev)
    RELX_CONFIG = $(CURDIR)/relx.dev.config
else ifeq ($(ENV),local)
    RELX_CONFIG = $(CURDIR)/relx.local.config
else
    RELX_CONFIG ?= $(CURDIR)/relx.config
endif

Q

消息確認機制QoS

https://blog.csdn.net/Jessechanrui/article/details/88399012

socket 數據粘包問題、拆包問題

消息投遞機制(未實現功能)

判斷用戶是否在線，如果用戶離線，直接存儲離線消息
用戶在線（or 用戶上線），判斷erlang is_process_alive(Pid) 馬上投遞一次
沒有收到消息之前， 2 S -> 5 S -> 7S -> 11 S 投遞4次
如果收到消息清理定時器，清理數據庫消息
4次投遞都未確認消息，待用戶下次登錄再投遞

erlang 的shell 訪問遠程節

erl -name debug@127.0.0.1
auth:set_cookie('imboy'),net_adm:ping('imboy@127.0.0.1').
net_adm:names().
{ok,[{"imboy",55042},{"debug",60595}]}

按 Ctrl+G 出现user switch command
然后输入

r 'imboy@127.0.0.1'

按回车

在按 J 机器显示节点:
 --> j
   1  {shell,start,[init]}
   2* {'imboy@127.0.0.1',shell,start,[]}

在 * 的就是默认的可连接节点，其中的1 行，就是你现在的master节点

按 c 就能连接

你如果要连接到第三节点的话，直接 输入 c 6 回车就行了。

chat_store_repo:lookup(1).

curl -L https://github.com/sile/erldash/releases/download/0.1.1/erldash-0.1.1.x86_64-unknown-linux-musl -o erldash
chmod +x erldash
./erldash imboy@127.0.0.1 -c imboy

webrtc

websocket 在線工具調試

為了簡化程式碼取消WS了在線調試（如有必要，以後可以看情況添加一個h5頁面做調試工具）

http://coolaf.com/tool/chattest io:format(“pn”, [token_ds:encrypt_token(4)]).

(imboy@127.0.0.1)10>  hashids_translator:uid_encode(4).
<<"8ybk5b">>
(imboy@127.0.0.1)11> hashids_translator:uid_encode(1).
<<"kybqdp">>
{"id":"text5","type":"C2C","from":"8ybk5b","to":"kybqdp","payload":{"msg_type":"text","text":"text5"},"created_at":1650118822382,"server_ts":1650118823376}

Email

gen_smtp_client:send({"sender@gmail.com", ["receiver@gmail.com"], "Subject: testing"},
   [{relay, "smtp.gmail.com"}, {ssl, true}, {username, "sender@gmail.com"},
      {password, "senderpassword"}]).

imboy_cache.erl

copy from https://github.com/zotonic/zotonic/blob/master/apps/zotonic_core/src/support/z_depcache.erl

為了項目風格統一，並且不依賴zotonic.hrl ，所以修改了module名稱
The Module name was changed in order to maintain a uniform project style and not rely on zotonic.hrl

imboy_cache:set(a, 1).
imboy_cache:get(a).
imboy_cache:memo(fun() ->
    a
end).

imboy_session

imboy_session:join(1, <<"ios">>, spawn(fun() -> receive _ -> ok end end), <<"did11">>).
imboy_session:join(1, <<"andriod">>, spawn(fun() -> receive _ -> ok end end), <<"did12">>).
imboy_session:join(1, <<"macos">>, self(), <<"3f039a2b4724a5b7">>).

imboy_session:join(2, <<"ios">>, spawn(fun() -> receive _ -> ok end end), <<"did21">>).
imboy_session:join(2, <<"andriod">>, spawn(fun() -> receive _ -> ok end end), <<"did22">>).
imboy_session:join(3, <<"andriod">>, spawn(fun() -> receive _ -> ok end end), <<"did32">>).

imboy_session:count().
imboy_session:count_user().
imboy_session:count_user(1).
imboy_session:list(1).

syn:

ets:select(syn_pg_by_name_chat, [{ '$1', [], ['$1']}]).
ets:select(syn_pg_by_name_chat, [{ '$1', [], ['$1']}], 10).

khepri

imboy_session:put(1, <<"ios">>, <<"did11">>, spawn(fun() -> receive _ -> ok end end)).
imboy_session:put(1, <<"andriod">>, <<"did12">>, spawn(fun() -> receive _ -> ok end end)).

imboy_session:put(2, <<"ios">>, <<"did21">>, spawn(fun() -> receive _ -> ok end end)).
imboy_session:put(2, <<"andriod">>, <<"did22">>, spawn(fun() -> receive _ -> ok end end)).


imboy_session:put(10000, <<"andriod">>, <<"did22">>, spawn(fun() -> receive _ -> ok end end)).

imboy_session:get(1).

How do I design the path to get the following data

https://github.com/rabbitmq/khepri/discussions/201

khepri:get({s, Uid})
{ok, [{1,<<"ios">>,<0.11996.0>,<<"did11">>}, {1,<<"andriod">>,<0.11820.1>,<<"did2">>}]}

*匹配任何樹節點並**匹配樹節點中的任何路徑

Uid1 = 1, Uid2 = 2, DType1 = <<"ios">>,  DType2 = <<"andriod">>.
DID1 = <<"did1">>, DID2 = <<"did2">>.
Pid1 = self(). Pid2 = self().

Uid1Bin = integer_to_binary(Uid1).

Path1 = <<"/:s/", Uid1Bin/binary, "https://gitee.com/", DType1/binary>>.
khepri:put(Path1, {Uid1, DType1, Pid1, DID1}).

Path2 = <<"/:s/", Uid1Bin/binary, "https://gitee.com/", DType2/binary>>.
khepri:put(Path2, {Uid1, DType2, Pid2, DID2}).

khepri:get_many("/:s/1/*").
khepri:get_many("/:s/1/**").


    {ok, Map1} = imboy_session:get(1),
    maps:fold(fun(K, Val, Acc) ->
        io:format("acc~p, k ~p, val: ~p,~n", [Acc, K, Val])
    end, [], Map1).



integer_to_list(4, 2).

khepri:delete_many("/:s/1/*").

eturnal

_build/product/rel/eturnal/bin/eturnal console

消息確認機制

服務端，發送消息程式碼如下：

    MillisecondList = [0, 1500, 1500, 3000, 1000, 3000, 5000],
    message_ds:send_next(Uid, DType, MsgId, Msg, MillisecondList),

MillisecondList 頻率控制列表，列表元素以單位為毫秒；
- 0 標識第1條消息馬上發送；
- 1500 表示，消息1500毫米內沒有ack，的情況下會再次投遞
- 其他邏輯元素邏輯如上
Uid 用戶ID
MsgId 消息ID，消息唯一標識
Msg 消息json文本

客戶端收到消息後發送以下文本數據(對erlang來說是binary文本)，用於消息收到確認

CLIENT_ACK,type,msgid,did

4段信息以半角逗號（ , ）分隔：

第1段： CLIENT_ACK 為固定消息確認前綴
第2段： type 是IM消息類型
第3段： msgid 消息唯一ID
第4段： did 登錄用戶設備ID

測試驗證數據，收到測試觀察，該消息確認機制有效

http://coolaf.com/tool/chattest

ws://192.168.1.4:9800/ws/?authorization=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2Njg1Njk3MTY0MDksInN1YiI6InRrIiwidWlkIjoiYnltajVnIn0.zPojzN6IfxzIfU4CCJodguaAMcGPDx3XLTvou6-U9A8

CLIENT_ACK,type,msgid,did

CurrentUid = 13238,
MsgId = <<"msgid">>,
DID = <<"did">>.
webrtc_ws_logic:event(13238, <<"ios">>, MsgId, <<"Msg bin">>).


http://coolaf.com/tool/chattest

// 1
ws://192.168.1.4:9800/ws/?authorization=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NjExMDM5NDQ5MTgsInN1YiI6InRrIiwidWlkIjoia3licWRwIn0.FYhYR0KzHZe9kHEeTbcYWwahyqLXBE7rUWaQgyI5I14

1 = imboy_hashids:uid_decode("kybqdp").
108 = imboy_hashids:uid_decode("7b4v1b").
{"id":"cdsgrbgppoodp0gvpb60","type":"C2C","from":"kybqdp","to":"7b4v1b","payload":{"msg_type":"text","text":"1to108"},"created_at":1668877742828}

108
ws://192.168.1.4:9800/ws/?authorization=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NjkwNDc2Nzc3MDgsInN1YiI6InRrIiwidWlkIjoiN2I0djFiIn0.n19M6-kR_p4EtqJMst4kO1cqgdG5F2gyNY6QL46xpR8
{"id":"cdsgrbgppoodp0gvpb61","type":"C2C","to":"kybqdp","from":"7b4v1b","payload":{"msg_type":"text","text":"108to1"},"created_at":1668877742828}

#使用erlang做後端flutter做前端開發的一款開源的即時聊天解決方案基於erlangotp的高性能web框架cowboy #做後端服務用8核16G #主機100萬PPS壓測保持100萬TCP穩定在線90分鐘以上 #支援基於webrtc #一對一視頻通話功能

使用erlang做後端、flutter做前端開發的一款開源的即時聊天解決方案（基於erlang/otp的高性能web框架cowboy 做後端服務，用”8核16G 主機（100萬PPS）”壓測，保持100萬+TCP穩定在線90分鐘以上） 支援基於webrtc 一對一視頻通話功能