柯里化是一种将使用多个参数的一个函数转换成一系列使用一个参数的函数的技术

1
2
3
4
5
6
7
8
9
10
function add(a, b) {
return a + b;
}

// 执行 add 函数,一次传入两个参数即可
add(1, 2) // 3

// 假设有一个 curry 函数可以做到柯里化
var addCurry = curry(add);
addCurry(1)(2) // 3

前置知识:JavaScript是静态作用域

闭包:访问自由变量的函数

1
2
3
4
5
6
7
var a = 1;//既不是foo的局部变量,也不是foo函数的参数,a为自由变量

function foo() {
console.log(a);
}

foo();//1

即使上下文被销毁,它仍然存在,因为在作用域链上被引用了,是js的一个特性,目前如PHP,Java不会原生支持

面试题

常见的新手面试题,我遇到过好几次(作用域+闭包考点)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
var data = [];

for (var i = 0; i < 3; i++) {
data[i] = function () {
console.log(i);
};
}

data[0]();
data[1]();
data[2]();


//closure
var data = [];

for (var i = 0; i < 3; i++) {
data[i] = (function (i) {
return function(){
console.log(i);
}
})(i);
}

data[0]();//不用找global的i
data[1]();
data[2]();

1.install remote ssh in vscode

2.click remote explorer and select ssh targets

3.click remote ssh configure or press F1 and input remote-ssh:Open configuration file

4.selete path ~/.ssh/config,and modify config file

if you dont have rsa ,please generate keys before

1
2
3
4
5
6
//optional
ssh-keygen
# passphrase can be empty and then generate keys in `~/.ssh`
# put *.pub (public key) to your server (~/.ssh/) and excute `cat id_rsa.pub >> authorized_keys` to merge Previous file
# now rsa keys are ready

1
2
3
4
5
6
7
Host alias
HostName 8.888.88.8
User root
IdentityFile ~/.ssh/id_rsa
RSAAuthentication yes
PubkeyAuthentication yes
PasswordAuthentication no
  • Host alias–>your remote server name
  • hostName–>server ip
  • User–>login username
  • IdentityFile–>private key path
  • RSAAuthentication–>optional
  • PubkeyAuthentication–>optional
  • PasswordAuthentication–>no password login

5.login without password ready

What

ECMAscript 6 原生提供了 Promise 对象。

Promise 对象代表了未来将要发生的事件,用来传递异步操作的消息。

Promise 对象有以下两个特点:

1、对象的状态不受外界影响。Promise 对象代表一个异步操作,有三种状态:

  • pending: 初始状态,不是成功或失败状态。
  • fulfilled: 意味着操作成功完成。
  • rejected: 意味着操作失败。

只有异步操作的结果,可以决定当前是哪一种状态,任何其他操作都无法改变这个状态。这也是 Promise 这个名字的由来,它的英语意思就是「承诺」,表示其他手段无法改变。

2、一旦状态改变,就不会再变,任何时候都可以得到这个结果。Promise 对象的状态改变,只有两种可能:从 Pending 变为 Resolved 和从 Pending 变为 Rejected。只要这两种情况发生,状态就凝固了,不会再变了,会一直保持这个结果。就算改变已经发生了,你再对 Promise 对象添加回调函数,也会立即得到这个结果。这与事件(Event)完全不同,事件的特点是,如果你错过了它,再去监听,是得不到结果的。

1
2
3
4
var promise = new Promise(function(resolve, reject) {
// 异步处理
// 处理结束后、调用resolve 或 reject
});

以上来自:菜鸟https://www.runoob.com/w3cnote/javascript-promise-object.html

Why

因为在2020年01月07日有一篇文章讲了使用promise实现延时队列的一道面试题,因为之前写业务没有用到过所以一直以为用处不大,但今天对接阿里的录音文件识别转文字的接口中,示例代码是一个setInterval轮询得到结果的一种方式,但是他带来了一个很严重的问题

!!没有办法返回前端转文字的结果!!

大概代码如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
//url:https://help.aliyun.com/document_detail/94242.html?spm=a2c4g.11174283.6.601.15eb7275a8rq00
// 这段代码会异步执行,可以得到结果,但是直接用这个代码返回给前端
client.submitTask(taskParams, options).then((response) => {
console.log(response);
// 服务端响应信息的状态描述StatusText。
var statusText = response.StatusText;
if (statusText != 'SUCCESS') {
console.log('录音文件识别请求响应失败!')
return;
}
console.log('录音文件识别请求响应成功!');
// 获取录音文件识别请求任务的TaskId,以供识别结果查询使用。
var taskId = response.TaskId;
/**
* 以TaskId为查询参数,提交识别结果查询请求。
* 以轮询的方式进行识别结果的查询,直到服务端返回的状态描述为"SUCCESS"、SUCCESS_WITH_NO_VALID_FRAGMENT,
* 或者为错误描述,则结束轮询。
*/
var taskIdParams = {
TaskId : taskId
};
var timer = setInterval(() => {
client.getTaskResult(taskIdParams).then((response) => {
console.log('识别结果查询响应:');
console.log(response);
var statusText = response.StatusText;
if (statusText == 'RUNNING' || statusText == 'QUEUEING') {
// 继续轮询,注意间隔周期。
}
else {
if (statusText == 'SUCCESS' || statusText == 'SUCCESS_WITH_NO_VALID_FRAGMENT') {
console.log('录音文件识别成功:');
var sentences = response.Result;
console.log(sentences);
}
else {
console.log('录音文件识别失败!');
}
// 退出轮询
clearInterval(timer);
}
}).catch((error) => {
console.error(error);
// 异常情况,退出轮询。
clearInterval(timer);
});
}, 10000);
}).catch((error) => {
console.error(error);
});
}

How

使用promise进行包裹,等到promise内部的函数取到了结果在返回

1
2
3
4
5
6
7
8
if (statusText == 'SUCCESS' || statusText == 'SUCCESS_WITH_NO_VALID_FRAGMENT') {
console.log('录音文件识别成功:');
var sentences = response.Result;
console.log(sentences);
//这里新增resolve
} else {
console.log('录音文件识别失败!');
}

外层通过如下代码实现

1
2
3
4
var promise = new Promise(function(resolve, reject) {
// 异步处理
// 处理结束后、调用resolve 或 reject
});
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
async function getWords() {
return new Promise((resolve, reject) => {
client
.submitTask(taskParams, options)
.then(response => {
console.log(response);
// 服务端响应信息的状态描述StatusText。
const statusText = response.StatusText;
if (statusText != 'SUCCESS') {
console.log('录音文件识别请求响应失败!');
}
console.log('录音文件识别请求响应成功!');
// 获取录音文件识别请求任务的TaskId,以供识别结果查询使用。
const taskId = response.TaskId;
/**
* 以TaskId为查询参数,提交识别结果查询请求。
* 以轮询的方式进行识别结果的查询,直到服务端返回的状态描述为"SUCCESS"、SUCCESS_WITH_NO_VALID_FRAGMENT,
* 或者为错误描述,则结束轮询。
*/
const taskIdParams = {
TaskId: taskId,
};

const timer = setInterval(() => {
client
.getTaskResult(taskIdParams)
.then(response => {
console.log('识别结果查询响应:');
console.log(response);
const statusText = response.StatusText;
if (statusText == 'RUNNING' || statusText == 'QUEUEING') {
// 继续轮询,注意间隔周期。
} else {
if (
statusText == 'SUCCESS' ||
statusText == 'SUCCESS_WITH_NO_VALID_FRAGMENT'
) {
console.log('录音文件识别成功:');
let sentences = '';
for (const s of response.Result.Sentences) {
sentences += s.Text;
}
console.log(response.Result);
resolve(sentences);//**重点**//
// return sentences;
} else {
console.log('录音文件识别失败!');
}
// 退出轮询
clearInterval(timer);
}
})
.catch(error => {
console.error(error);
// 异常情况,退出轮询。
clearInterval(timer);
});
}, 10000);
})
.catch(error => {
console.error(error);
});
});
}
return await getWords();//返回前端,翻译结果

另外记录一件事情,左侧单元图标地址:https://fontawesome.com/v4.7.0/icons/

what

Jupiter = Julia + Python + R

Jupyter notebook(http://jupyter.org/) 是一种 Web 应用,能让用户将说明文本、数学方程、代码和可视化内容全部组合到一个易于共享的文档中。

why

  • 将代码和文档结合在一起,更直观的编写人工智能,大数据的代码
  • 分块运行
  • 直接运行shell不需要切换环境
  • so on

how

  1. Download images
1
2
$docker pull jupyter jupyter/scipy-notebook:latest
$docker run -itd --rm -p 1000:8888 -e JUPYTER_ENABLE_LAB=yes -v /home/zehai/jupyter:/home/jovyan/work --name jupyter jupyter/scipy-notebook:latest
  1. docker logs -f container’s ID and find token
1
2
3
4
5
To access the notebook, open this file in a browser:
file:///home/jovyan/.local/share/jupyter/runtime/nbserver-6-open.html
Or copy and paste one of these URLs:
http://896bb1e66101:8888/?token=fda8565a9b5cd5b8c621b45322ee72f716fd7ddea089fb51
or http://127.0.0.1:8888/?token=fda8565a9b5cd5b8c621b45322ee72f716fd7ddea089fb51
  1. more info visit official docs
  2. enjoy (pics powered by cherevero)

3318a2fadaf085f2bee7f0de3b42971c.png

what

To solve some problems

  • some web only use markdown and can’t upload pictures,such as v2ex.com
  • some pics you don’t want to give it to others for long time,such as your interesting story
  • give your blog’s can speed when download bigger pics
  • and so on

Picture Bed can offer you a excellent platform to share your pictures and protect them, however it has a problem that you need a server to run the service,even though you can use 七牛云,alioss,weibo for free.

why

Chevereto is aim what I find

  • dockerhub has chevereto images
  • Combined with ShareX (only for windows😢),chevereto can write markdown essay easily
  • it has api ,you can make it stronger
  • Chevereto Free v1.2.2 now
  • Something others you can discover by yourself

how

Chevereto is a php project , I use docker to run it

1
2
3
4
5
6
7
8
9
10
docker pull nmtan/chevereto:latest

//use docker-compose.yml(next block)

// or docker run
docker run -it --name chevereto -d -p 8000:80 -v "/home/xxx/images":/var/www/html/images -e "CHEVERETO_DB_HOST=127.0.0.1" -e "CHEVERETO_DB_USERNAME=root" -e "CHEVERETO_DB_PASSWORD=rootpass" -e "CHEVERETO_DB_NAME=chevereto" -e "CHEVERETO_DB_PREFIX=chv_" nmtan/chevereto
//-v save photos in server instead of container
//-e mysql:5.7.31 host,username,password,db_name(db must exist first)
//open chrome and input 127.0.0.1:8000

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
//this is docker-compose.yml
version: '3'

services:
db:
image: mariadb
volumes:
- ./database:/var/lib/mysql:rw
restart: always
networks:
- private
environment:
MYSQL_ROOT_PASSWORD: xxxxx
MYSQL_DATABASE: xxxxx
MYSQL_USER: xxxxx
MYSQL_PASSWORD: xxxxx

chevereto:
depends_on:
- db
image: nmtan/chevereto
restart: always
networks:
- private
environment:
CHEVERETO_DB_HOST: db
CHEVERETO_DB_USERNAME: xxxxxx
CHEVERETO_DB_PASSWORD: xxxxx
CHEVERETO_DB_NAME: xxxxx
CHEVERETO_DB_PREFIX: chv_
volumes:
- ./images:/var/www/html/images:rw
- ./php.ini:/usr/local/etc/php/php.ini:ro
ports:
- 8080:80

networks:
private:


// start command
nohup docker-compose up &> run.log &
disown

You maybe run into a stone wall when you first visit 127.0.0.1:8000

  • Before you can use docker exec -it chevereto bash into container /var/www/html

  • no permission write phots to /home/xxx/images,you can use chmod -R 777 /home/xxx/images

  • no permission update chevereto from 1.1.4 to1.2.2 ,no update possible: /app/install/update/temp/ path,that is no temp folder in /app/install/update/ under version 1.2.0,you can mkdir temp and then chmod -R 777 ./temp and then refresh the webpage ,the prics bed will update successfully

So , you can use ip address to visit your chevereto . However , we usually use domain name such as example.com to visit web, a https isn ecessary as well

  • 1.Use aliyun to apply a free ssl license for a domain name such as pics.example.com
  • 2.Download pem and keys to your server and put it in nginx conf folder
  • 3.Use the conf as follows
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
server {
listen 80;
server_name pics.example.com;
return 301 https://pics.example.com$request_uri;
}

server {
listen 443 ssl;
server_name pics.example.com;
gzip on;

ssl_certificate cert/xxxxxx9_pics.example.com.pem; # pem's filename
ssl_certificate_key cert/xxxxxx9_pics.example.com.key;# key's filename

location / {
proxy_redirect off;
proxy_pass http://dockername;

proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Ssl on;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;

client_max_body_size 100m;
client_body_buffer_size 128k;

proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
}
}
    1. And then you can visit https://pics.example.com

That‘s my story that building pics bed ,and hope to help you.

2020-09-28 append

use picgo,upload pictures in typora to chevereto

  1. GitHub download picgo,mac use .dmp,and then install it

  2. open 插件设置,search chevereto and install chevereto 1.0.0

  3. Open 图床设置>Chevereto Uploader and put in params,

    Url is your upload service ip/domain

    Key is chevereto api in Dashboard>Settings>API>API v1 key

    param is not in use now

1
2
Url:https://example.com/api/1/upload
Key:xxx
  1. Click 确定 and 设为默认图库
  2. make sure server is on PicGo设置>设置Server>点击设置,if it is on ,noting should be done

and then we modify config in typaro

  1. Open Typora and open ‘preferences>Images`
  2. Choose Upload images in when Insert and check apply above rules to local images and apply above rules to online images in option, and I suggest you check both of them to approve all pics managed by chevereto
  3. Choose PicGo.app in images Uploader and click Test Uploader to test your upload pictures automatically

for more information ,youcan visit

thanks

what

MQ-message queue

三足鼎立

rocketmq​​ -Made by Java 吞吐量高一些 阿里中间件

rabbitmq -Made by Erlang

Kafka-

以后有更多的了解再补充性能/功能差距

why

功能:解耦(双方通过mq交互)、异步、削峰

应用:

  • 阿里双11

问题:

  • 处理好新增的复杂性
  • 处理好系统可用性

how

之所以选择rabbitmq是因为rocketmq的nameserver所需要的内存太大了,更何况boker,对于1C2G的乞丐机器来说根本跑不起来

1.docker run

Because of rocketmq need more than

1
2
3
4
5
6
7
8
9
docker pull rabbitmq:management

docker run -dit --name rabbitmq -e RABBITMQ_DEFAULT_USER=admin -e RABBITMQ_DEFAULT_PASS=admin -p 15672:15672 -p 5672:5672 rabbitmq:management

--name containername
-e RABBITMQ_DEFAULT_USER 参数用户名,密码同理
-p 端口映射,主机:容器,15672-UI,5672-service
rabbitmq:management image's name

2.Usage

1.open chrome and input ‘localhost:15672’ or ‘192.168.1.1:15672’ then you can touch rabbitmq UI

Overview–the queued msg, msg rate in your server, some global counts, your nodes stats (if u use the above method,you only see one node in the screen ),you also can build a cluster with more nodes

Connections–

Channels–

Exchanges–direct,fanout,headers,match,trace,topic

Queses–

Admin–users management with passport && permission

2.use 5672 in your code

1
2
3
4
5
6
7
8
amqp.connect({
protocol: 'amqp',
hostname: 'example.com',//localhost
port: '5672',
username: 'admin',
password: 'xxx',
vhost: '/',//important
})

more in official docs–> I’m doc

or some blogs–>I’m blog

or my GitHub–>click here

eventLoop

之前也有过章节
node整理
Node.js

有看到石墨技术文档

cnode技术文档,作者:youth7

记录以下知识点:

  • nodejs的event是基于libuv浏览器的event loop则在html5的规范中明确定义,两个事物有明显的区别
  • process.nextTick()在6个阶段结束的时候都会执行

eventLoop

timers 执行setTimeout()setInterval()中到期的callback
I/O callbacks 上一轮循环中有少数的I/Ocallback会被延迟到这一轮的这一阶段执行
idle, prepare 仅内部使用
poll 最为重要的阶段,执行I/O callback,在适当的条件下会阻塞在这个阶段
check 执行setImmediate的callback
close callbacks 执行close事件的callback,例如socket.on("close",func)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
   ┌───────────────────────┐
┌─>│ timers │
│ └──────────┬────────────┘
│ ┌──────────┴────────────┐
│ │ I/O callbacks │
│ └──────────┬────────────┘
│ ┌──────────┴────────────┐
│ │ idle, prepare │
│ └──────────┬────────────┘ ┌───────────────┐
│ ┌──────────┴────────────┐ │ incoming: │
│ │ poll │<─────┤ connections, │
│ └──────────┬────────────┘ │ data, etc. │
│ ┌──────────┴────────────┐ └───────────────┘
│ │ check │
│ └──────────┬────────────┘
│ ┌──────────┴────────────┐
└──┤ close callbacks │
└───────────────────────┘
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
# /deps/uv/src/unix/core.c
int uv_run(uv_loop_t* loop, uv_run_mode mode) {
int timeout;
int r;
int ran_pending;

r = uv__loop_alive(loop);
// if(uv_has_active_hanles||uv_has_active_reqs || lop->closing_handles!=NULL)retrun true
if (!r)
uv__update_time(loop);

while (r != 0 && loop->stop_flag == 0) {
uv__update_time(loop);
// main
uv__run_timers(loop);//timer phase
ran_pending = uv__run_pending(loop);//IO callback pharse
uv__run_idle(loop);//idle phase
uv__run_prepare(loop);// prepare phase
// main end

timeout = 0;
if ((mode == UV_RUN_ONCE && !ran_pending) || mode == UV_RUN_DEFAULT)
timeout = uv_backend_timeout(loop);

uv__io_poll(loop, timeout);//poll phase
uv__run_check(loop);//check phase
uv__run_closing_handles(loop);//closing pharse

if (mode == UV_RUN_ONCE) {
/* UV_RUN_ONCE implies forward progress: at least one callback must have
* been invoked when it returns. uv__io_poll() can return without doing
* I/O (meaning: no callbacks) when its timeout expires - which means we
* have pending timers that satisfy the forward progress constraint.
*
* UV_RUN_NOWAIT makes no guarantees about progress so it's omitted from
* the check.
*/
// UV_RUN_ONCE 至少有一个回调执行,不然该循环就空转了,满足前进要求
// 这也是[文章](https://zehai.info/2020/04/10/2020-04-10-eventloop/)中写到:
// poll为空,eventloop将检查timer是否有快到的,如果需要执行,eventloop将要进入timers阶段来顺序执行timer callback
uv__update_time(loop);
uv__run_timers(loop);
}

r = uv__loop_alive(loop);
if (mode == UV_RUN_ONCE || mode == UV_RUN_NOWAIT)
break;
}

/* The if statement lets gcc compile it to a conditional store. Avoids
* dirtying a cache line.
*/
if (loop->stop_flag != 0)
loop->stop_flag = 0;

return r;
}

timers phase

执行setTimeout()setInterval()中到期的callback

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
void uv__run_timers(uv_loop_t* loop) {
struct heap_node* heap_node;
uv_timer_t* handle;

for (;;) {
heap_node = heap_min(timer_heap(loop));
if (heap_node == NULL)
break;
// 取出堆中最快要被执行的timer
// #define container_of(ptr, type, member)
// ((type *) ((char *) (ptr) - offsetof(type, member)))
// 没看懂 handle是怎么生成的
handle = container_of(heap_node, uv_timer_t, heap_node);
if (handle->timeout > loop->time)//执行时间大于eventloop循环一次时间,退出phase下次再说
break;

uv_timer_stop(handle);// remove handle
uv_timer_again(handle);// 多次重复的timer再塞进去
handle->timer_cb(handle);// invoke callback
}
}

I/O callbacks

上一轮循环中有少数的I/Ocallback会被延迟到这一轮的这一阶段执行

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
//deps/uv/src/unix/core.c
static int uv__run_pending(uv_loop_t* loop) {
QUEUE* q;
QUEUE pq;
uv__io_t* w;

if (QUEUE_EMPTY(&loop->pending_queue))//isEmpty
return 0;

QUEUE_MOVE(&loop->pending_queue, &pq);//move

while (!QUEUE_EMPTY(&pq)) {
q = QUEUE_HEAD(&pq);//find
QUEUE_REMOVE(q);//pop
QUEUE_INIT(q);
w = QUEUE_DATA(q, uv__io_t, pending_queue);
w->cb(loop, w, POLLOUT);//unitl queue empty
}

return 1;
}

Idle and prepare phase

/* loop */

void uv__run_idle(uv_loop_t* loop);

void uv__run_check(uv_loop_t* loop);

void uv__run_prepare(uv_loop_t* loop);

1
2
3
4
5
6
7
8
9
10
11
12
13
void uv__run_##name(uv_loop_t* loop) {
uv_##name##_t* h;
QUEUE queue;
QUEUE* q;
QUEUE_MOVE(&loop->name##_handles, &queue);//QUEUE_MOVE
while (!QUEUE_EMPTY(&queue)) {//util empty
q = QUEUE_HEAD(&queue);//pop
h = QUEUE_DATA(q, uv_##name##_t, queue);//element->handle
QUEUE_REMOVE(q);//remove
QUEUE_INSERT_TAIL(&loop->name##_handles, q);//insert tail
h->name##_cb(h);//callback
}
}

!!!poll phase!!!

最为重要的阶段,执行I/O callback,在适当的条件下会阻塞在这个阶段

可见poll阶段的任务就是阻塞等待监听的事件来临,然后执行对应的callback,其中阻塞是带有超时时间的,以下几种情况都会使得超时时间为0

  • uv_run处于UV_RUN_NOWAIT模式下
  • uv_stop()被调用
  • 没有活跃的handles和request
  • 有活跃的idle handles
  • 有等待关闭的handles

如果上述都不符合,则超时时间为距离现在最近的timer;如果没有timer则poll阶段会一直阻塞下去

个人理解nodejs的服务,大部分时间会被阻塞在这个阶段,而不去执行closing

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
// 不行了,看不懂了
void uv__io_poll(uv_loop_t* loop, int timeout) {
struct pollfd events[1024];
struct pollfd pqry;
struct pollfd* pe;
struct poll_ctl pc;
QUEUE* q;
uv__io_t* w;
uint64_t base;
uint64_t diff;
int have_signals;
int nevents;
int count;
int nfds;
int i;
int rc;
int add_failed;

if (loop->nfds == 0) {
assert(QUEUE_EMPTY(&loop->watcher_queue));
return;
}

while (!QUEUE_EMPTY(&loop->watcher_queue)) {//until watcher queue empty
q = QUEUE_HEAD(&loop->watcher_queue);
QUEUE_REMOVE(q);
QUEUE_INIT(q);

w = QUEUE_DATA(q, uv__io_t, watcher_queue);
assert(w->pevents != 0);
assert(w->fd >= 0);
assert(w->fd < (int) loop->nwatchers);

pc.events = w->pevents;
pc.fd = w->fd;

add_failed = 0;
if (w->events == 0) {
pc.cmd = PS_ADD;
if (pollset_ctl(loop->backend_fd, &pc, 1)) {
if (errno != EINVAL) {
assert(0 && "Failed to add file descriptor (pc.fd) to pollset");
abort();
}
/* Check if the fd is already in the pollset */
pqry.fd = pc.fd;
rc = pollset_query(loop->backend_fd, &pqry);
switch (rc) {
case -1:
assert(0 && "Failed to query pollset for file descriptor");
abort();
case 0:
assert(0 && "Pollset does not contain file descriptor");
abort();
}
/* If we got here then the pollset already contained the file descriptor even though
* we didn't think it should. This probably shouldn't happen, but we can continue. */
add_failed = 1;
}
}
if (w->events != 0 || add_failed) {
/* Modify, potentially removing events -- need to delete then add.
* Could maybe mod if we knew for sure no events are removed, but
* content of w->events is handled above as not reliable (falls back)
* so may require a pollset_query() which would have to be pretty cheap
* compared to a PS_DELETE to be worth optimizing. Alternatively, could
* lazily remove events, squelching them in the mean time. */
pc.cmd = PS_DELETE;
if (pollset_ctl(loop->backend_fd, &pc, 1)) {
assert(0 && "Failed to delete file descriptor (pc.fd) from pollset");
abort();
}
pc.cmd = PS_ADD;
if (pollset_ctl(loop->backend_fd, &pc, 1)) {
assert(0 && "Failed to add file descriptor (pc.fd) to pollset");
abort();
}
}

w->events = w->pevents;
}

assert(timeout >= -1);
base = loop->time;
count = 48; /* Benchmarks suggest this gives the best throughput. */

for (;;) {
nfds = pollset_poll(loop->backend_fd,
events,
ARRAY_SIZE(events),
timeout);

/* Update loop->time unconditionally. It's tempting to skip the update when
* timeout == 0 (i.e. non-blocking poll) but there is no guarantee that the
* operating system didn't reschedule our process while in the syscall.
*/
SAVE_ERRNO(uv__update_time(loop));

if (nfds == 0) {
assert(timeout != -1);
return;
}

if (nfds == -1) {
if (errno != EINTR) {
abort();
}

if (timeout == -1)
continue;

if (timeout == 0)
return;

/* Interrupted by a signal. Update timeout and poll again. */
goto update_timeout;
}

have_signals = 0;
nevents = 0;

assert(loop->watchers != NULL);
loop->watchers[loop->nwatchers] = (void*) events;
loop->watchers[loop->nwatchers + 1] = (void*) (uintptr_t) nfds;

for (i = 0; i < nfds; i++) {
pe = events + i;
pc.cmd = PS_DELETE;
pc.fd = pe->fd;

/* Skip invalidated events, see uv__platform_invalidate_fd */
if (pc.fd == -1)
continue;

assert(pc.fd >= 0);
assert((unsigned) pc.fd < loop->nwatchers);

w = loop->watchers[pc.fd];

if (w == NULL) {
/* File descriptor that we've stopped watching, disarm it.
*
* Ignore all errors because we may be racing with another thread
* when the file descriptor is closed.
*/
pollset_ctl(loop->backend_fd, &pc, 1);
continue;
}

/* Run signal watchers last. This also affects child process watchers
* because those are implemented in terms of signal watchers.
*/
if (w == &loop->signal_io_watcher)
have_signals = 1;
else
w->cb(loop, w, pe->revents);

nevents++;
}

if (have_signals != 0)
loop->signal_io_watcher.cb(loop, &loop->signal_io_watcher, POLLIN);

loop->watchers[loop->nwatchers] = NULL;
loop->watchers[loop->nwatchers + 1] = NULL;

if (have_signals != 0)
return; /* Event loop should cycle now so don't poll again. */

if (nevents != 0) {
if (nfds == ARRAY_SIZE(events) && --count != 0) {
/* Poll for more events but don't block this time. */
timeout = 0;
continue;
}
return;
}

if (timeout == 0)
return;

if (timeout == -1)
continue;

update_timeout:
assert(timeout > 0);

diff = loop->time - base;
if (diff >= (uint64_t) timeout)
return;

timeout -= diff;
}
}

check phase

见idle prepare

close

关闭handle

1
2
3
4
5
6
7
8
9
10
11
12
13
static void uv__run_closing_handles(uv_loop_t* loop) {
uv_handle_t* p;
uv_handle_t* q;

p = loop->closing_handles;
loop->closing_handles = NULL;

while (p) {
q = p->next_closing;
uv__finish_close(p);
p = q;
}
}

where is process.nextTick

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
//lib/internal/process/task_queues.js
// `nextTick()` will not enqueue any callback when the process is about to
// exit since the callback would not have a chance to be executed.
// 意思就是nextTick在进程快要结束时不会排队callback,因为没有机会执行
// 你们看引用的文档吧,我看不下去了😭
// 主要的思路是JS执行process.nexTick(),然后将callback交给c++执行
function nextTick(callback) {
if (typeof callback !== 'function')
throw new ERR_INVALID_CALLBACK(callback);

if (process._exiting)
return;

let args;
switch (arguments.length) {
case 1: break;
case 2: args = [arguments[1]]; break;
case 3: args = [arguments[1], arguments[2]]; break;
case 4: args = [arguments[1], arguments[2], arguments[3]]; break;
default:
args = new Array(arguments.length - 1);
for (let i = 1; i < arguments.length; i++)
args[i - 1] = arguments[i];
}

if (queue.isEmpty())
setHasTickScheduled(true);
const asyncId = newAsyncId();
const triggerAsyncId = getDefaultTriggerAsyncId();
const tickObject = {
[async_id_symbol]: asyncId,
[trigger_async_id_symbol]: triggerAsyncId,
callback,
args
};
if (initHooksExist())
emitInit(asyncId, 'TickObject', triggerAsyncId, tickObject);
queue.push(tickObject);//封装callback push
//进入c
}

question

1.setTimeout vs setImmediate

  • phase执行顺序
  • expire设置0是不是立刻执行
1
2
3
4
5
6
7
setTimeout(() => {
console.log('setTimeout')
}, 0)

setImmediate(() => {
console.log('setImmediate')
})
  • setTimeout/setInterval 的第二个参数取值范围是:[1, 2^31 - 1],如果超过这个范围则会初始化为 1,即 setTimeout(fn, 0) === setTimeout(fn, 1)。
  • setTimeout 的回调函数在 timer 阶段执行,setImmediate 的回调函数在 check 阶段执行,event loop 的开始会先检查 timer 阶段,但是在开始之前到 timer 阶段会消耗一定时间,所以就会出现两种情况:
    • timer 前的准备时间超过 1ms,满足 loop->time >= 1,则执行 timer 阶段(setTimeout)的回调函数
    • timer 前的准备时间小于 1ms,则先执行 check 阶段(setImmediate)的回调函数,下一次 event loop 执行 timer 阶段(setTimeout)的回调函数

在举例:

1
2
3
4
5
6
7
8
9
10
setTimeout(() => {
console.log('setTimeout')
}, 0)

setImmediate(() => {
console.log('setImmediate')
})

const start = Date.now()
while (Date.now() - start < 10);//准备时间超过1ms,则直接执行timer

2.setTimeout vs setImmediate 2

1
2
3
4
5
6
7
8
9
10
11
12
13
const fs = require('fs')

fs.readFile(__filename, () => {
setTimeout(() => {
console.log('setTimeout')
}, 0)

setImmediate(() => {
console.log('setImmediate')
})
})
//setImmediate
//setTimeout

在引用一下官方对于check phase的介绍

This phase allows a person to execute callbacks immediately after the poll phase has completed. If the poll phase becomes idle and scripts have been queued with setImmediate(), the event loop may continue to the check phase rather than waiting.

setImmediate() is actually a special timer that runs in a separate phase of the event loop. It uses a libuv API that schedules callbacks to execute after the poll phase has completed.

Generally, as the code is executed, the event loop will eventually hit the poll phase where it will wait for an incoming connection, request, etc. However, if a callback has been scheduled with setImmediate() and the poll phase becomes idle, it will end and continue to the check phase rather than waiting for poll events.

fs.readFile 的回调函数执行完后:

  1. 注册 setTimeout 的回调函数到 timer 阶段
  2. 注册 setImmediate 的回调函数到 check 阶段
  3. event loop 从 pool 阶段出来继续往下一个阶段执行,恰好是 check 阶段,所以 setImmediate 的回调函数先执行
  4. 本次 event loop 结束后,进入下一次 event loop,执行 setTimeout 的回调函数

所以,在 I/O Callbacks 中注册的 setTimeout 和 setImmediate,永远都是 setImmediate 先执行。

3.process.nextTick()

1
2
3
4
5
6
7
8
9
10
11
12
setInterval(() => {
console.log('setInterval')
}, 100)

process.nextTick(function tick () {
process.nextTick(tick)
})
//note
setImmediate(function immediate () {
console.log('111');//会直接打印出很多次111
setImmediate(immediate)
})

运行结果:setInterval 永远不会打印出来

//这个在node官方文档也有相关的描述

//我在这里也进行了笔记记录

//允许用户处理errors,清理不需要的资源,事件循环前 尝试重新连接

//有时有必要在eventloop继续之前,在call stack unwound之后,让callback执行

解释:process.nextTick 会无限循环,将 event loop 阻塞在 microtask 阶段,导致 event loop 上其他 macrotask 阶段的回调函数没有机会执行。//这段解释是前端的,后端是没有microtask的实际队列的

解决方法通常是用 setImmediate 替代 process.nextTick,如下:

1
2
3
4
5
6
7
setInterval(() => {
console.log('setInterval')
}, 100)

setImmediate(function immediate () {
setImmediate(immediate)
})

运行结果:每 100ms 打印一次 setInterval。

解释:process.nextTick 内执行 process.nextTick 仍然将 tick 函数注册到当前 microtask 的尾部,所以导致 microtask 永远执行不完; setImmediate 内执行 setImmediate 会将 immediate 函数注册到下一次 event loop 的 check 阶段,而不是当前正在执行的 check 阶段,所以给了 event loop 上其他 macrotask 执行的机会。

再看个例子:

1
2
3
4
5
6
7
8
9
10
11
12
13
setImmediate(() => {
console.log('setImmediate1')
setImmediate(() => {
console.log('setImmediate2')
})
process.nextTick(() => {
console.log('nextTick')
})
})

setImmediate(() => {
console.log('setImmediate3')
})

运行结果:

1
2
3
4
setImmediate1
setImmediate3
nextTick
setImmediate2

注意:并不是说 setImmediate 可以完全替代 process.nextTick,process.nextTick 在特定场景下还是无法被替代的,比如我们就想将一些操作放到最近的 microtask 里执行。

4.promise

1
2
3
4
5
const promise = Promise.resolve()
.then(() => {
return promise
})
promise.catch(console.error)

运行结果:

1
2
3
4
5
6
TypeError: Chaining cycle detected for promise #<Promise>
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7)
at Function.Module.runMain (module.js:667:11)
at startup (bootstrap_node.js:187:16)
at bootstrap_node.js:607:3

解释:promise.then 类似于 process.nextTick,都会将回调函数注册到 microtask 阶段。上面代码会导致死循环,类似前面提到的:

1
2
3
process.nextTick(function tick () {
process.nextTick(tick)
})

再看个例子:

1
2
3
4
5
6
7
8
9
const promise = Promise.resolve()

promise.then(() => {
console.log('promise')
})

process.nextTick(() => {
console.log('nextTick')
})

运行结果:

1
2
nextTick
promise

解释:promise.then 虽然和 process.nextTick 一样,都将回调函数注册到 microtask,但优先级不一样。process.nextTick 的 microtask queue 总是优先于 promise 的 microtask queue 执行。

5.promise执行顺序

1
2
3
4
5
6
7
8
9
10
11
12
13
14
setTimeout(() => {
console.log(1)
}, 0)

new Promise((resolve, reject) => {
console.log(2)
for (let i = 0; i < 10000; i++) {
i === 9999 && resolve()
}
console.log(3)
}).then(() => {
console.log(4)
})
console.log(5)

运行结果:

1
2
3
4
5
2
3
5
4
1

解释:Promise 构造函数是同步执行的,所以先打印 2、3,然后打印 5,接下来 event loop 进入执行 microtask 阶段,执行 promise.then 的回调函数打印出 4,然后执行下一个 macrotask,恰好是 timer 阶段的 setTimeout 的回调函数,打印出 1。

6.综合

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
setImmediate(() => {
console.log(1)
setTimeout(() => {
console.log(2)
}, 100)
setImmediate(() => {
console.log(3)
})
process.nextTick(() => {
console.log(4)
})
})
process.nextTick(() => {
console.log(5)
setTimeout(() => {
console.log(6)
}, 100)
setImmediate(() => {
console.log(7)
})
process.nextTick(() => {
console.log(8)
})
})
console.log(9)

运行结果:

1
2
3
4
5
6
7
8
9
9
5
8
1
7
4
3
6
2

process.nextTick、setTimeout 和 setImmediate 的组合,请读者自己推理吧。

other source code

setTimeout()

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
//lib/timers/promises.js
//setTimeout(function(){},expire)
function setTimeout(after, value, options = {}) {
const args = value !== undefined ? [value] : value;
if (options == null || typeof options !== 'object') {
return PromiseReject(
new ERR_INVALID_ARG_TYPE(
'options',
'Object',
options));
}
const { signal, ref = true } = options;
if (signal !== undefined &&
(signal === null ||
typeof signal !== 'object' ||
!('aborted' in signal))) {
return PromiseReject(
new ERR_INVALID_ARG_TYPE(
'options.signal',
'AbortSignal',
signal));
}
if (typeof ref !== 'boolean') {
return PromiseReject(
new ERR_INVALID_ARG_TYPE(
'options.ref',
'boolean',
ref));
}
// TODO(@jasnell): If a decision is made that this cannot be backported
// to 12.x, then this can be converted to use optional chaining to
// simplify the check.
if (signal && signal.aborted)
return PromiseReject(lazyDOMException('AbortError'));
return new Promise((resolve, reject) => {
const timeout = new Timeout(resolve, after, args, false, true);
if (!ref) timeout.unref();
insert(timeout, timeout._idleTimeout);
if (signal) {
signal.addEventListener('abort', () => {
if (!timeout._destroyed) {
// eslint-disable-next-line no-undef
clearTimeout(timeout);
reject(lazyDOMException('AbortError'));
}
}, { once: true });
}
});
}

1. 算法简介

1.1 二分

Why: 复杂度O(n)—>O(logn)

使用限制:有序数组

1.2 大O表示

指出算法运行时间的增速,算法需要做的就是把O(n^2)优化到O(n)等

2. 选择排序

2.1 数组 链表

数组:连续物理空间,可随机访问,增删数据复杂度高

链表:分散无力空间,不可随机访问(只能顺序),增删数据复杂度低

数组 链表
读改 O(1) O(n)
增删 O(n) O(1)

根据互相特性,选择合适的方式,如频繁增删用链表,反之用数组

2.2 选择排序

复杂度:O(n^2)

遍历n 个元素选择 最小/大的,遍历n-1个元素选择 最小/大的

3. 递归

类比:套娃 :call_me_hand:

性能和易读不可兼得

避免死循环

尾递归可以解决部分性能问题

递归调用栈是性能降低的原因,遵循FIFO

4. 快排

核心:分而治之divide and conquer,快排只是其中的一个应用

思想:递归的一种应用

快排(递归)是一种函数式编程

快排通过基准值(可以选第一个元素)进行分而治之

5. 散列表

实现方式:数组,非链表,检索值key类似数组的下表,可直接访问value

应用:DNS,阻止重复数据(类set集),作缓存(服务器端)

复杂度 散列平均 散列最糟 数组 链表
查找 1 n 1 n
插入删除 1 n n 1

装填因子(0.4)=散列元素(4)/位置总数(10)

避免冲突:1.良好的散列函数(均匀分布) 2.较低的装填因子(<0.7

将满时候:1.申请两倍于原来的 新空间 2.hash所有元素到新空间

冲突解决:

  • 开放地址(最简单就是冲突顺延下一位,直到为空)
  • 拉链发(指在某个位子上再拉一条链表,非👖拉链)

6.BFS

广度优先搜索breadth first search,解决无加权最短路径问题之一

应用:国际跳棋,拼写检查,人际关系网络

7. Dijkstra

正加权有向无环图的解决算法

  1. 最短时间内到达的节点
  2. 更新该节点临接节点的开销
  3. 重复
  4. 计算最终路径

解决环:

负加权:bellman ford algorithm

8. Greedy

每步最优–>全局最优,得到近似正确的结果

9.DP

列出所有可能

10. K最邻近算法

11.next

树解决了二分查找中,插入删除O(n)降低到O(log n),但是降低了随机访问能力

树包括:二叉树,平衡二叉树,B树 B+树,,红黑树

反向索引:散列表,用于创建搜索引擎—>应用:傅里叶变换

并行算法,单机并行or分布式,应用:mapreduce,map->映射 ,reduce->归并

布隆过滤器:庞大的散列表(如谷歌的上亿条),通常使用redis实现,是一种概率型数据结构(偶尔出错),使用理由,存储空间少

hyperLogLog:类似布隆,是个日志

SHA算法

  • 散列的一种应用
  • 判断两个(超大)文件是否相同(散列值相同)
  • SHA(用户输入密码)?== 数据库存储的SHA值,且拖库后无法还原密码
  • SHA是一系列算法的统称,包括SHA-0 ,SHA-1 SHA-2 SHA-3 bcrypt etc
  • SHA全局敏感(改动局部,整体全变),SIMhash局部敏感(局部改变,散列值局部改变),后者用于判断网页是否已经搜集,作业是否抄袭,相似度查询
  • diffie-hellman密钥交换
    • 双方无需知道加密算法,破解难度大
    • 公钥与私钥,client获取公钥后,1.使用公钥加密 2.服务器端使用私钥解密

线性规划:simplex算法

0%