-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.xml
More file actions
741 lines (705 loc) · 106 KB
/
index.xml
File metadata and controls
741 lines (705 loc) · 106 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>我的精神家园</title>
<link>https://MoonerHigh.github.io/</link>
<description>Recent content on 我的精神家园</description>
<generator>Hugo -- gohugo.io</generator>
<language>en-us</language>
<lastBuildDate>Mon, 13 Feb 2023 00:44:00 +0800</lastBuildDate><atom:link href="https://MoonerHigh.github.io/index.xml" rel="self" type="application/rss+xml" /><item>
<title>快速了解CDN</title>
<link>https://MoonerHigh.github.io/p/%E5%BF%AB%E9%80%9F%E4%BA%86%E8%A7%A3cdn/</link>
<pubDate>Mon, 13 Feb 2023 00:44:00 +0800</pubDate>
<guid>https://MoonerHigh.github.io/p/%E5%BF%AB%E9%80%9F%E4%BA%86%E8%A7%A3cdn/</guid>
<description><p>CDN的全称为Content Delivery Network,即内容分发网络。其基本思路是尽可能避开互联网上有可能影响数据传输速度和稳定性的瓶颈和环节,使内容传输得更快、更稳定。通过在网络各处放置节点服务器所构成的在现有的互联网基础之上的一层智能虚拟网络,CDN系统能够实时地根据网络流量和各节点的连接、负载状况以及到用户的距离和响应时间等综合信息将用户的请求重新导向离用户最近的服务节点上。其目的是使用户可就近取得所需内容,解决 Internet网络拥挤的状况,提高用户访问网站的响应速度。</p>
<blockquote>
<p>简单来说,CDN通过在网络节点各处放置边缘机房,当用户请求时使用负载均衡技术使用户可以就近取得内容,从而解决网络拥挤的状况。</p>
</blockquote>
<h2 id="cdn的实现原理">CDN的实现原理</h2>
<blockquote>
<p>💡 CDN的基本原理为反向代理。反向代理是指以代理服务器来接受Internet上的连接请求,然后将请求转发给内部网络上的服务器, 并将服务器上的请求结果通过代理服务器返回给请求连接的的客户端。此时,代理服务器的表现为一个节点服务器,通过部署大量的代理服务器,就可以达到多节点CDN的效果。</p>
</blockquote>
<p><img src="https://pic.imgdb.cn/item/63e90c4f4757feff33c353a4.png"
loading="lazy"
alt="CDN.png"
></p>
<p>当用户在浏览器中输入域名进行访问时,会被DNS服务器进行解析,域名在被解析之后,会指向CDN域名服务器,CDN域名服务器这个时候会返回域名的CNAME记录,然后再对CNAME记录进行解析,这个时候会访问负载均衡服务器得到缓存服务器的地址,最后再去对IP进行访问从而得到数据并返回数据给浏览器回显。</p>
<h2 id="cdn解决了什么问题">CDN解决了什么问题?</h2>
<ol>
<li>使用大量边缘服务器通过负载均衡减少网络拥堵</li>
<li>降低服务器带宽成本</li>
<li>当源服务器宕机时,CDN缓存服务器可以提供服务</li>
<li>压缩静态内容,从而提高访问速度</li>
</ol>
</description>
</item>
<item>
<title>快速了解TCP协议</title>
<link>https://MoonerHigh.github.io/p/%E5%BF%AB%E9%80%9F%E4%BA%86%E8%A7%A3tcp%E5%8D%8F%E8%AE%AE/</link>
<pubDate>Fri, 10 Feb 2023 22:32:46 +0800</pubDate>
<guid>https://MoonerHigh.github.io/p/%E5%BF%AB%E9%80%9F%E4%BA%86%E8%A7%A3tcp%E5%8D%8F%E8%AE%AE/</guid>
<description><blockquote>
<p>💡 TCP/IP(Transmission Control Protocol/Internet Protocol,传输控制协议/网际协议)是指能够在多个不同网络间实现信息传输的协议簇。TCP/IP协议不仅仅指的是<a class="link" href="https://baike.baidu.com/item/TCP/33012?fromModule=lemma_inlink" target="_blank" rel="noopener"
>TCP</a> 和<a class="link" href="https://baike.baidu.com/item/IP/224599?fromModule=lemma_inlink" target="_blank" rel="noopener"
>IP</a>两个协议,而是指一个由<a class="link" href="https://baike.baidu.com/item/FTP/13839?fromModule=lemma_inlink" target="_blank" rel="noopener"
>FTP</a>、<a class="link" href="https://baike.baidu.com/item/SMTP/175887?fromModule=lemma_inlink" target="_blank" rel="noopener"
>SMTP</a>、TCP、<a class="link" href="https://baike.baidu.com/item/UDP/571511?fromModule=lemma_inlink" target="_blank" rel="noopener"
>UDP</a>、IP等协议构成的协议簇, 只是因为在TCP/IP协议中TCP协议和IP协议最具代表性,所以被称为TCP/IP协议。</p>
</blockquote>
<h3 id="tcpip分层管理">TCP/IP分层管理</h3>
<p>TCP/IP协议族按层次分可以分为以下四层。</p>
<blockquote>
<p>应用层-传输层-网络层-数据链路层</p>
</blockquote>
<p>TCP/IP分层的好处是可以解耦,如果互联网只由一个协议统筹,某个地方需要改变设计时,需要把整体都替换掉,而分层之后只需要把变动的层替换掉即可。</p>
<p><img src="https://pic.imgdb.cn/item/63e50b984757feff33ce609e.jpg"
loading="lazy"
alt="OSI-TCP_IP.jpeg"
></p>
<h3 id="tcpip通信传输流">TCP/IP通信传输流</h3>
<p>利用TCP/IP协议簇进行通信时,会通过分层顺序与对方进行确认,下面我们用HTTP来举例说明一下。</p>
<ol>
<li>客户端作为发送端在应用层使用HTTP协议向某一个web页面发送请求。</li>
<li>为了传输方便,在传输层(TCP协议)把从应用层接收到的数据(HTTP请求报文)进行分割,并在各个报文上打上标记序号及端口号后转发给网络层。</li>
<li>在网络层(IP协议),增加作为通信地址的MAC地址然后转发给数据链路层。</li>
</ol>
<p>当接收端的服务器在链路层收到数据,按序往上层发送,一直到应用层,才算能真正接收到客户端发来的HTTP请求。</p>
<p>发送端在层与层之间传输数据时,每经过一层必定会被打上该层所属的首部信息。反之,接收端在层与层传输数据时,每经过一层必会把对应的首部消除。</p>
<h3 id="tcp三次握手">TCP三次握手</h3>
<blockquote>
<p>最开始的时候客户端和服务端都是处于CLOSED状态,主动打开的是客户端,被动打开的是服务端。</p>
</blockquote>
<p><img src="https://img-blog.csdnimg.cn/114bd9b53c554319ae0db15b57b472b8.gif#pic_center"
loading="lazy"
alt="https://img-blog.csdnimg.cn/114bd9b53c554319ae0db15b57b472b8.gif#pic_center"
></p>
<ol>
<li>TCP服务器先创建传输控制块TCB,时刻准备接受客户进程的连接请求,此时服务器就进入了LISTEN状态;</li>
<li>TCP客户进程也是先创建传输控制块TCB,然后向服务器发出连接请求报文,这是报文首部中的同步为SYN=1,同时选择一个初始序列号seq=x,此时客户端进程进入了SYN-SENT(同步到已发送)状态。TCP规定,SYN报文段(SYN=1的报文段)不能携带数据,但需要消耗掉一个序号。</li>
<li>TCP服务器收到请求报文后,如果同意连接,则发出确认报文。确认报文中应该ACK=1,SYN=1,确认号是ack=x+1,同时也要初始化一个序列号seq=y,此时TCP服务器进程进入了SYN-RCVD状态。这个报文也不能携带数据,但是同样要消耗一个序号。</li>
<li>TCP客户端收到确认后,还要向服务器给出确认。确认报文的ACK=1,ack=y+1,自己的序列号seq=x+1,此时TCP连接建立,客户端进入ESTABLISHED(已建立连接)状态。TCP规定,ACK报文段可以携带数据,但是如果不携带数据则不消耗信号。</li>
<li>当服务器收到客户端的确认后也进入ESTABLISHED状态,此时双方就可以开始通信了。</li>
</ol>
<p><strong><strong>为什么TCP客户端最后还要发送一次确认呢?</strong></strong></p>
<blockquote>
<p>主要防止已经失效的连接请求报文突然又传送到了服务器,从而产生错误。</p>
</blockquote>
<h3 id="tcp的释放-四次挥手">TCP的释放-四次挥手</h3>
<p><img src="https://img-blog.csdnimg.cn/0c17d72769aa4bd2b95bbaee1749b39d.gif#pic_center"
loading="lazy"
alt="https://img-blog.csdnimg.cn/0c17d72769aa4bd2b95bbaee1749b39d.gif#pic_center"
></p>
<blockquote>
<p>数据传输完毕后,双方都可释放连接。最开始的时候,客户端和服务器都是处于ESTABLISHED状态,然后客户端主动关闭,服务器被动关闭。</p>
</blockquote>
<ul>
<li>
<p>参考资料</p>
<p><a class="link" href="https://weread.qq.com/web/bookDetail/3da32b505dd9f43da9a1aca" target="_blank" rel="noopener"
>图解HTTP-上野宣-微信读书</a></p>
<p><a class="link" href="https://blog.csdn.net/qzcsu/article/details/72861891" target="_blank" rel="noopener"
>两张动图-彻底明白TCP的三次握手与四次挥手_小书go的博客-CSDN博客_tcp三次握手</a></p>
</li>
</ul>
</description>
</item>
<item>
<title>搜狗爬虫</title>
<link>https://MoonerHigh.github.io/p/%E6%90%9C%E7%8B%97%E7%88%AC%E8%99%AB/</link>
<pubDate>Wed, 01 Feb 2023 01:32:50 +0800</pubDate>
<guid>https://MoonerHigh.github.io/p/%E6%90%9C%E7%8B%97%E7%88%AC%E8%99%AB/</guid>
<description><p>分享之前接单私活写的一个爬虫单,客户要求爬取微信公众号的标题,所属公众号,文章发布时间,文章地址,代码本身并不复杂,直接采用搜狗引擎里的微信公众号接口即可。</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span><span class="lnt">38
</span><span class="lnt">39
</span><span class="lnt">40
</span><span class="lnt">41
</span><span class="lnt">42
</span><span class="lnt">43
</span><span class="lnt">44
</span><span class="lnt">45
</span><span class="lnt">46
</span><span class="lnt">47
</span><span class="lnt">48
</span><span class="lnt">49
</span><span class="lnt">50
</span><span class="lnt">51
</span><span class="lnt">52
</span><span class="lnt">53
</span><span class="lnt">54
</span><span class="lnt">55
</span><span class="lnt">56
</span><span class="lnt">57
</span><span class="lnt">58
</span><span class="lnt">59
</span><span class="lnt">60
</span><span class="lnt">61
</span><span class="lnt">62
</span><span class="lnt">63
</span><span class="lnt">64
</span><span class="lnt">65
</span><span class="lnt">66
</span><span class="lnt">67
</span><span class="lnt">68
</span><span class="lnt">69
</span><span class="lnt">70
</span><span class="lnt">71
</span><span class="lnt">72
</span><span class="lnt">73
</span><span class="lnt">74
</span><span class="lnt">75
</span><span class="lnt">76
</span><span class="lnt">77
</span><span class="lnt">78
</span><span class="lnt">79
</span><span class="lnt">80
</span><span class="lnt">81
</span><span class="lnt">82
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">time</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">urllib.parse</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">urllib.request</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">random</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">requests</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">parsel</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">csv</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">tqdm</span> <span class="kn">import</span> <span class="n">tqdm</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># 爬取内容写入csv文件</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">download_csv</span><span class="p">(</span><span class="n">title_str</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">otherStyleTime</span><span class="p">,</span> <span class="n">article_url</span><span class="p">):</span>
</span></span><span class="line"><span class="cl"> <span class="n">dit</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="s1">&#39;标题&#39;</span><span class="p">:</span> <span class="n">title_str</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="s1">&#39;公众号&#39;</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="s1">&#39;文章发布时间&#39;</span><span class="p">:</span> <span class="n">otherStyleTime</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="s1">&#39;文章地址&#39;</span><span class="p">:</span> <span class="n">article_url</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="p">}</span>
</span></span><span class="line"><span class="cl"> <span class="n">csv_writer</span><span class="o">.</span><span class="n">writerow</span><span class="p">(</span><span class="n">dit</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="c1"># print(title_str, name, otherStyleTime, article_url)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># 解析响应内容</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">parse_lable</span><span class="p">(</span><span class="n">response</span><span class="p">):</span>
</span></span><span class="line"><span class="cl"> <span class="n">selector</span> <span class="o">=</span> <span class="n">parsel</span><span class="o">.</span><span class="n">Selector</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">text</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">lis</span> <span class="o">=</span> <span class="n">selector</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;.news-list li&#39;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="n">li</span> <span class="ow">in</span> <span class="n">lis</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">title_list</span> <span class="o">=</span> <span class="n">li</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;.txt-box h3 a::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"> <span class="n">num</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">title_list</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">num</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">title_str</span> <span class="o">=</span> <span class="n">keyword</span> <span class="o">+</span> <span class="n">title_list</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"> <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">title_str</span> <span class="o">=</span> <span class="n">keyword</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">title_list</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="n">href</span> <span class="o">=</span> <span class="n">li</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;.txt-box h3 a::attr(href)&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"> <span class="n">article_url</span> <span class="o">=</span> <span class="s1">&#39;https://weixin.sogou.com&#39;</span> <span class="o">+</span> <span class="n">href</span>
</span></span><span class="line"><span class="cl"> <span class="n">name</span> <span class="o">=</span> <span class="n">li</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;.s-p a::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"> <span class="n">date</span> <span class="o">=</span> <span class="n">li</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;.s-p::attr(t)&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"> <span class="n">timeArray</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">localtime</span><span class="p">(</span><span class="nb">int</span><span class="p">(</span><span class="n">date</span><span class="p">))</span>
</span></span><span class="line"><span class="cl"> <span class="n">otherStyleTime</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s2">&#34;%Y-%m-</span><span class="si">%d</span><span class="s2"> %H:%M:%S&#34;</span><span class="p">,</span> <span class="n">timeArray</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">download_csv</span><span class="p">(</span><span class="n">title_str</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">otherStyleTime</span><span class="p">,</span> <span class="n">article_url</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># 访问搜狗视频获取最新Cookie</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">get_new_cookies</span><span class="p">():</span>
</span></span><span class="line"><span class="cl"> <span class="c1"># 搜狗视频url</span>
</span></span><span class="line"><span class="cl"> <span class="n">url</span> <span class="o">=</span> <span class="s1">&#39;https://v.sogou.com/v?ie=utf8&amp;query=&amp;p=40030600&#39;</span>
</span></span><span class="line"><span class="cl"> <span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;User-Agent&#39;</span><span class="p">:</span> <span class="s1">&#39;Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36&#39;</span><span class="p">}</span>
</span></span><span class="line"><span class="cl"> <span class="n">rst</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span><span class="o">=</span><span class="n">url</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">allow_redirects</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">cookies</span> <span class="o">=</span> <span class="n">rst</span><span class="o">.</span><span class="n">cookies</span><span class="o">.</span><span class="n">get_dict</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">cookies</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">&#39;SNUID&#39;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">&#39;__main__&#39;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">keyword</span> <span class="o">=</span> <span class="nb">input</span><span class="p">(</span><span class="s2">&#34;请输入检索内容:&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">f</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;公众号文章_&#39;</span> <span class="o">+</span> <span class="n">keyword</span> <span class="o">+</span> <span class="s1">&#39;.csv&#39;</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="s1">&#39;a&#39;</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">&#39;utf-8&#39;</span><span class="p">,</span> <span class="n">newline</span><span class="o">=</span><span class="s1">&#39;&#39;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">csv_writer</span> <span class="o">=</span> <span class="n">csv</span><span class="o">.</span><span class="n">DictWriter</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">fieldnames</span><span class="o">=</span><span class="p">[</span><span class="s1">&#39;标题&#39;</span><span class="p">,</span> <span class="s1">&#39;公众号&#39;</span><span class="p">,</span> <span class="s1">&#39;文章发布时间&#39;</span><span class="p">,</span> <span class="s1">&#39;文章地址&#39;</span><span class="p">])</span>
</span></span><span class="line"><span class="cl"> <span class="n">csv_writer</span><span class="o">.</span><span class="n">writeheader</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"> <span class="n">SNUID</span><span class="o">=</span><span class="n">get_new_cookies</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"> <span class="n">headers</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="s1">&#39;Cookie&#39;</span><span class="p">:</span> <span class="sa">f</span><span class="s1">&#39;IPLOC=CN5101; SUID=A08DB0753822910A0000000062D4E15C; SUV=1658118495598633; &#39;</span>
</span></span><span class="line"><span class="cl"> <span class="sa">f</span><span class="s1">&#39;ppinf=5|1658121514|1659331114&#39;</span>
</span></span><span class="line"><span class="cl"> <span class="sa">f</span><span class="s1">&#39;|dHJ1c3Q6MToxfGNsaWVudGlkOjQ6MjAxN3x1bmlxbmFtZTozNjolRTglODclQUElRTUlOUMlQTglRTclOEIlQUMlRTglQTElOEN8Y3J0OjEwOjE2NTgxMjE1MTR8cmVmbmljazozNjolRTglODclQUElRTUlOUMlQTglRTclOEIlQUMlRTglQTElOEN8dXNlcmlkOjQ0Om85dDJsdU5QS01HS01jX2o5WjM5aG5XY1I4d29Ad2VpeGluLnNvaHUuY29tfA; pprdig=IjtTyvBfu3HPN0JuCq-GC-ua97U2qrVaL1lKmQcS5ISR8XkUoZxSWujpBlyP0zSHSItfzFGPWnzTuIhBHYtUwY-PI6COy-6I8RcER_mopugTqcBlwdO-Sc_p7j8a51jQaet93CEJ-MdpiSceBG_Nmu5cLTPN5XhC_3B92CSovak; ppinfo=80456da5d9; passport=5|1658121514|1659331114|dHJ1c3Q6MToxfGNsaWVudGlkOjQ6MjAxN3x1bmlxbmFtZTozNjolRTglODclQUElRTUlOUMlQTglRTclOEIlQUMlRTglQTElOEN8Y3J0OjEwOjE2NTgxMjE1MTR8cmVmbmljazozNjolRTglODclQUElRTUlOUMlQTglRTclOEIlQUMlRTglQTElOEN8dXNlcmlkOjQ0Om85dDJsdU5QS01HS01jX2o5WjM5aG5XY1I4d29Ad2VpeGluLnNvaHUuY29tfA|242a929d54|IjtTyvBfu3HPN0JuCq-GC-ua97U2qrVaL1lKmQcS5ISR8XkUoZxSWujpBlyP0zSHSItfzFGPWnzTuIhBHYtUwY-PI6COy-6I8RcER_mopugTqcBlwdO-Sc_p7j8a51jQaet93CEJ-MdpiSceBG_Nmu5cLTPN5XhC_3B92CSovak; sgid=29-55943661-AWLU7Sr6zRRQxuX98MZMHMw; wuid=1658155657696; VIDEO_DEBUG=off; SNUID=</span><span class="si">{</span><span class="n">SNUID</span><span class="si">}</span><span class="s1">; ariaDefaultTheme=undefined; ppmdig=16581727390000000ba8e97bdbf22bfcefd4d50f43d86a61&#39;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="s1">&#39;Host&#39;</span><span class="p">:</span> <span class="s1">&#39;weixin.sogou.com&#39;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="s1">&#39;User-Agent&#39;</span><span class="p">:</span> <span class="s1">&#39;Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 &#39;</span>
</span></span><span class="line"><span class="cl"> <span class="s1">&#39;Safari/537.36&#39;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="p">}</span>
</span></span><span class="line"><span class="cl"> <span class="c1"># proxies = {</span>
</span></span><span class="line"><span class="cl"> <span class="c1"># &#39;http&#39;:&#39;http://113.222.29.169:22008&#39;,</span>
</span></span><span class="line"><span class="cl"> <span class="c1"># &#39;https&#39;: &#39;https://113.222.29.169:22008&#39;,</span>
</span></span><span class="line"><span class="cl"> <span class="c1"># }</span>
</span></span><span class="line"><span class="cl"> <span class="nb">print</span><span class="p">(</span><span class="s1">&#39;-------------开始爬取-------------&#39;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="n">page</span> <span class="ow">in</span> <span class="n">tqdm</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">1000</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)):</span>
</span></span><span class="line"><span class="cl"> <span class="n">url</span> <span class="o">=</span> <span class="sa">f</span><span class="s1">&#39;https://weixin.sogou.com/weixin?query=</span><span class="si">{</span><span class="n">urllib</span><span class="o">.</span><span class="n">parse</span><span class="o">.</span><span class="n">quote</span><span class="p">(</span><span class="n">keyword</span><span class="p">)</span><span class="si">}</span><span class="s1">&amp;_sug_type_=&amp;s_from=input&amp;_sug_=n&#39;</span> \
</span></span><span class="line"><span class="cl"> <span class="sa">f</span><span class="s1">&#39;&amp;type=2&amp;page=</span><span class="si">{</span><span class="n">page</span><span class="si">}</span><span class="s1">&amp;ie=utf8 &#39;</span>
</span></span><span class="line"><span class="cl"> <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span><span class="o">=</span><span class="n">url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">parse_lable</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="c1"># 控制爬取频率</span>
</span></span><span class="line"><span class="cl"> <span class="c1"># time.sleep(random.randint(8, 10))</span>
</span></span><span class="line"><span class="cl"> <span class="k">except</span> <span class="ne">KeyboardInterrupt</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="nb">print</span><span class="p">(</span><span class="s1">&#39;</span><span class="se">\n</span><span class="s1">-----------------操作中止---------------------&#39;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="nb">print</span><span class="p">(</span><span class="s1">&#39;-------------爬取完成------------&#39;</span><span class="p">)</span>
</span></span></code></pre></td></tr></table>
</div>
</div><p>代码本身很简单,关键在于搜狗的反爬机制,Cookie很容易失效,可能爬两次程序就挂了,客户自己没有办法调试。</p>
<p>那么直接看Cookie这一项,他是由多个key-value拼接成的,经过解析,发现起决定性作用的只有key为SUID的这一项。</p>
<p>现在只需要直接获取最新的Cookie拿到SUID的value拼装成一个新Cookie,定制其成为新的请求头就可以再去访问页面就可以了。</p>
<p>至于新的Cookie怎么来…..</p>
<p>解决方法其实很简单,搜狗引擎提供了很多接口,都是兄弟产品,我们现在操作的是weixin.sougou.com,试着访问其他接口,比如搜狗视频v.sogou.com,发现Cookie的规则是一致的,但是关键字的值并不一样,我只需要访问url再取其Cookie就可以了。对搜狗视频的访问操作不足以触发其反爬机制,这就形成了一个闭环。</p>
<p>最终的方案是在程序启动时,访问搜狗视频接口,拿到新的Cookie补充给headers里的Cookie,定制一个全新的请求头,再去循环爬取操作,最终写入文件。</p>
<p>其他代码都是孰能生巧,解决问题思路很重要。</p>
</description>
</item>
<item>
<title>New or Make</title>
<link>https://MoonerHigh.github.io/p/new-or-make/</link>
<pubDate>Sun, 29 Jan 2023 12:04:17 +0800</pubDate>
<guid>https://MoonerHigh.github.io/p/new-or-make/</guid>
<description><h2 id="new和make的区别">New和Make的区别</h2>
<p>以下是Go1.19中对new()和make()函数的解释:</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// The new built-in function allocates memory. The first argument is a type,
</span></span></span><span class="line"><span class="cl"><span class="c1">// not a value, and the value returned is a pointer to a newly
</span></span></span><span class="line"><span class="cl"><span class="c1">// allocated zero value of that type.
</span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="nb">new</span><span class="p">(</span><span class="nx">Type</span><span class="p">)</span> <span class="o">*</span><span class="nx">Type</span>
</span></span></code></pre></td></tr></table>
</div>
</div><p>new()是一个新的分配内存的函数,第一个参数是类型,不是值,返回的值是指向该类型零值的指针。</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// The make built-in function allocates and initializes an object of type
</span></span></span><span class="line"><span class="cl"><span class="c1">// slice, map, or chan (only). Like new, the first argument is a type, not a
</span></span></span><span class="line"><span class="cl"><span class="c1">// value. Unlike new, make&#39;s return type is the same as the type of its
</span></span></span><span class="line"><span class="cl"><span class="c1">// argument, not a pointer to it. The specification of the result depends on
</span></span></span><span class="line"><span class="cl"><span class="c1">// the type:
</span></span></span><span class="line"><span class="cl"><span class="c1">//
</span></span></span><span class="line"><span class="cl"><span class="c1">// Slice: The size specifies the length. The capacity of the slice is
</span></span></span><span class="line"><span class="cl"><span class="c1">// equal to its length. A second integer argument may be provided to
</span></span></span><span class="line"><span class="cl"><span class="c1">// specify a different capacity; it must be no smaller than the
</span></span></span><span class="line"><span class="cl"><span class="c1">// length. For example, make([]int, 0, 10) allocates an underlying array
</span></span></span><span class="line"><span class="cl"><span class="c1">// of size 10 and returns a slice of length 0 and capacity 10 that is
</span></span></span><span class="line"><span class="cl"><span class="c1">// backed by this underlying array.
</span></span></span><span class="line"><span class="cl"><span class="c1">// Map: An empty map is allocated with enough space to hold the
</span></span></span><span class="line"><span class="cl"><span class="c1">// specified number of elements. The size may be omitted, in which case
</span></span></span><span class="line"><span class="cl"><span class="c1">// a small starting size is allocated.
</span></span></span><span class="line"><span class="cl"><span class="c1">// Channel: The channel&#39;s buffer is initialized with the specified
</span></span></span><span class="line"><span class="cl"><span class="c1">// buffer capacity. If zero, or the size is omitted, the channel is
</span></span></span><span class="line"><span class="cl"><span class="c1">// unbuffered.
</span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="nb">make</span><span class="p">(</span><span class="nx">t</span> <span class="nx">Type</span><span class="p">,</span> <span class="nx">size</span> <span class="o">...</span><span class="nx">IntegerType</span><span class="p">)</span> <span class="nx">Type</span>
</span></span></code></pre></td></tr></table>
</div>
</div><p>make()函数用于初始化slice,map和channel。和new()比较像的是第一个参数也是一个类型,不是值。区别于new的地方在于make()返回的值和他参数类型是一样的。第二个类型size是一个可变长类型的参数,作用如下:</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"> <span class="nx">m</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">map</span><span class="p">[</span><span class="kt">int</span><span class="p">]</span><span class="kt">string</span><span class="p">,</span> <span class="mi">5</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="nx">s</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">([]</span><span class="kt">int</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="nx">channel</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="kt">string</span><span class="p">,</span> <span class="mi">5</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="nx">m</span><span class="p">,</span> <span class="nx">s</span><span class="p">,</span> <span class="nx">channel</span><span class="p">)</span>
</span></span></code></pre></td></tr></table>
</div>
</div><p>初始化一个slice,传入第一个参数为类型,第二个参数为长度,第三个参数为容量,容量不可以小于长度。初始化map的可变参数只有容量cap,而初始化channel的可变参数为缓冲大小。</p>
</description>
</item>
<item>
<title>逃逸分析</title>
<link>https://MoonerHigh.github.io/p/%E9%80%83%E9%80%B8%E5%88%86%E6%9E%90/</link>
<pubDate>Fri, 27 Jan 2023 13:58:43 +0800</pubDate>
<guid>https://MoonerHigh.github.io/p/%E9%80%83%E9%80%B8%E5%88%86%E6%9E%90/</guid>
<description><h2 id="堆内存与栈内存">堆内存与栈内存</h2>
<p>程序会在两个地方为变量分配内存,一个是全局的堆空间用来动态分配内存,另一个是goroutine的栈空间,Go语言拥有自动的垃圾回收机制,我们不需要关心内存到底是分配在堆上还是栈上,但是从性能的角度出发,分配在栈空间还是堆空间差异较大。</p>
<p>在函数中申请一个对象,如果分配在栈上,函数执行完毕自动回收,如果分配在堆上,则在函数执行完毕后某个时间点进行垃圾回收。</p>
<p>在栈上分配或者回收内存的开销很低,只需要两个CPU指令,一个是Push,一个是Pop,Push代表分配内存,Pop代表释放内存。在栈上分配内存,消耗的仅是将内存分配到栈上的时间,内存的IO速度通常能够达到30GB/s,因此在栈上分配内存效率比较高。</p>
<p>在堆上分配内存,一个很大的额外开销是垃圾回收,Go语言使用的是清除标记算法,并在此基础上,使用了三色标记法和写屏障算法,提高了效率。</p>
<h2 id="逃逸分析">逃逸分析</h2>
<p>Go语言中,堆内存是通过垃圾回收机制自动管理的,那么Go的编译器如何知道内存是分配在栈上还是堆上呢?编译器决定内存分配位置的方式,就叫逃逸分析。逃逸分析由编译器完成,作用于编译阶段。</p>
<h3 id="指针逃逸">指针逃逸</h3>
<p>在函数中创建了一个对象,返回了这个对象的指针。这种情况下,函数虽然退出了,但是因为指针的存在,对象的内存不能随着函数的结束而回收,因此只能分配在堆上。</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span> <span class="nx">main</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="s">&#34;fmt&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">Demo</span> <span class="kd">struct</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">name</span> <span class="kt">string</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">createDemo</span><span class="p">(</span><span class="nx">name</span> <span class="kt">string</span><span class="p">)</span> <span class="o">*</span><span class="nx">Demo</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">d</span> <span class="o">:=</span> <span class="nb">new</span><span class="p">(</span><span class="nx">Demo</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="nx">d</span><span class="p">.</span><span class="nx">name</span> <span class="p">=</span> <span class="nx">name</span>
</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">d</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">demo</span> <span class="o">:=</span> <span class="nf">createDemo</span><span class="p">(</span><span class="s">&#34;demo&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="nx">demo</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></td></tr></table>
</div>
</div><div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="k">go</span> <span class="nx">run</span> <span class="o">-</span><span class="nx">gcflags</span> <span class="s">&#34;-m -l&#34;</span> <span class="nx">Demo30</span><span class="p">.</span><span class="k">go</span>
</span></span></code></pre></td></tr></table>
</div>
</div><div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="p">.</span><span class="o">/</span><span class="nx">Demo30</span><span class="p">.</span><span class="k">go</span><span class="p">:</span><span class="mi">9</span><span class="p">:</span><span class="mi">17</span><span class="p">:</span> <span class="nx">leaking</span> <span class="nx">param</span><span class="p">:</span> <span class="nx">name</span>
</span></span><span class="line"><span class="cl"><span class="p">.</span><span class="o">/</span><span class="nx">Demo30</span><span class="p">.</span><span class="k">go</span><span class="p">:</span><span class="mi">10</span><span class="p">:</span><span class="mi">10</span><span class="p">:</span> <span class="nb">new</span><span class="p">(</span><span class="nx">Demo</span><span class="p">)</span> <span class="nx">escapes</span> <span class="nx">to</span> <span class="nx">heap</span>
</span></span><span class="line"><span class="cl"><span class="p">.</span><span class="o">/</span><span class="nx">Demo30</span><span class="p">.</span><span class="k">go</span><span class="p">:</span><span class="mi">16</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span> <span class="o">...</span> <span class="nx">argument</span> <span class="nx">does</span> <span class="nx">not</span> <span class="nx">escape</span>
</span></span><span class="line"><span class="cl"><span class="o">&amp;</span><span class="p">{</span><span class="nx">demo</span><span class="p">}</span>
</span></span></code></pre></td></tr></table>
</div>
</div><p>new(Demo) escapes to heap说明new(Demo)逃逸到了堆上</p>
<h3 id="interface动态类型逃逸">Interface动态类型逃逸</h3>
<p>空接口Interface{}可以表示任何类型,如果函数的参数为interface{},编译期无法确定具体类型,则会发生逃逸。</p>
<p>在上面的例子中</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">demo</span> <span class="o">:=</span> <span class="nf">createDemo</span><span class="p">(</span><span class="s">&#34;demo&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="nx">demo</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></td></tr></table>
</div>
</div><p>我们将main函数中的局部变量demo作为实际参数传给fmt.Println方法,</p>
<p>fmt.Println方法的实现如下</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">Println</span><span class="p">(</span><span class="nx">a</span> <span class="o">...</span><span class="nx">any</span><span class="p">)</span> <span class="p">(</span><span class="nx">n</span> <span class="kt">int</span><span class="p">,</span> <span class="nx">err</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nf">Fprintln</span><span class="p">(</span><span class="nx">os</span><span class="p">.</span><span class="nx">Stdout</span><span class="p">,</span> <span class="nx">a</span><span class="o">...</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></td></tr></table>
</div>
</div><p>参数类型为a…any即interface{}类型,因此发生了逃逸。</p>
<h3 id="栈空间不足">栈空间不足</h3>
<p>超过一定大小的局部变量会逃逸到堆上,变量大小不确定,也会逃逸到堆上。</p>
<h3 id="闭包">闭包</h3>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span> <span class="nx">main</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="s">&#34;fmt&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">Increment</span><span class="p">()</span> <span class="kd">func</span><span class="p">()</span> <span class="kt">int</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">n</span> <span class="o">:=</span> <span class="mi">0</span>
</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kd">func</span><span class="p">()</span> <span class="kt">int</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">n</span><span class="o">++</span>
</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">n</span>
</span></span><span class="line"><span class="cl"> <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">increment</span> <span class="o">:=</span> <span class="nf">Increment</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"> <span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="nf">increment</span><span class="p">())</span>
</span></span><span class="line"><span class="cl"> <span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="nf">increment</span><span class="p">())</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></td></tr></table>
</div>
</div><p>Increment()返回值是一个闭包函数,该匿名函数访问了外部变量n,外部变量n将一直存在,直到increment被销毁。变量n的内存不能随着函数Increment()推出而被销毁,因此将会逃逸到堆上。</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="err">➜</span> <span class="nx">Demo</span> <span class="k">go</span> <span class="nx">run</span> <span class="o">-</span><span class="nx">gcflags</span><span class="p">=</span><span class="o">-</span><span class="nx">m</span> <span class="nx">Demo31</span><span class="p">.</span><span class="k">go</span>
</span></span><span class="line"><span class="cl"><span class="err">#</span> <span class="nx">command</span><span class="o">-</span><span class="nx">line</span><span class="o">-</span><span class="nx">arguments</span>
</span></span><span class="line"><span class="cl"><span class="p">.</span><span class="o">/</span><span class="nx">Demo31</span><span class="p">.</span><span class="k">go</span><span class="p">:</span><span class="mi">5</span><span class="p">:</span><span class="mi">6</span><span class="p">:</span> <span class="nx">can</span> <span class="nx">inline</span> <span class="nx">Increment</span>
</span></span><span class="line"><span class="cl"><span class="p">.</span><span class="o">/</span><span class="nx">Demo31</span><span class="p">.</span><span class="k">go</span><span class="p">:</span><span class="mi">7</span><span class="p">:</span><span class="mi">9</span><span class="p">:</span> <span class="nx">can</span> <span class="nx">inline</span> <span class="nx">Increment</span><span class="p">.</span><span class="nx">func1</span>
</span></span><span class="line"><span class="cl"><span class="p">.</span><span class="o">/</span><span class="nx">Demo31</span><span class="p">.</span><span class="k">go</span><span class="p">:</span><span class="mi">14</span><span class="p">:</span><span class="mi">24</span><span class="p">:</span> <span class="nx">inlining</span> <span class="nx">call</span> <span class="nx">to</span> <span class="nx">Increment</span>
</span></span><span class="line"><span class="cl"><span class="p">.</span><span class="o">/</span><span class="nx">Demo31</span><span class="p">.</span><span class="k">go</span><span class="p">:</span><span class="mi">7</span><span class="p">:</span><span class="mi">9</span><span class="p">:</span> <span class="nx">can</span> <span class="nx">inline</span> <span class="nx">main</span><span class="p">.</span><span class="nx">func1</span>
</span></span><span class="line"><span class="cl"><span class="p">.</span><span class="o">/</span><span class="nx">Demo31</span><span class="p">.</span><span class="k">go</span><span class="p">:</span><span class="mi">15</span><span class="p">:</span><span class="mi">23</span><span class="p">:</span> <span class="nx">inlining</span> <span class="nx">call</span> <span class="nx">to</span> <span class="nx">main</span><span class="p">.</span><span class="nx">func1</span>
</span></span><span class="line"><span class="cl"><span class="p">.</span><span class="o">/</span><span class="nx">Demo31</span><span class="p">.</span><span class="k">go</span><span class="p">:</span><span class="mi">15</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span> <span class="nx">inlining</span> <span class="nx">call</span> <span class="nx">to</span> <span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span>
</span></span><span class="line"><span class="cl"><span class="p">.</span><span class="o">/</span><span class="nx">Demo31</span><span class="p">.</span><span class="k">go</span><span class="p">:</span><span class="mi">16</span><span class="p">:</span><span class="mi">23</span><span class="p">:</span> <span class="nx">inlining</span> <span class="nx">call</span> <span class="nx">to</span> <span class="nx">main</span><span class="p">.</span><span class="nx">func1</span>
</span></span><span class="line"><span class="cl"><span class="p">.</span><span class="o">/</span><span class="nx">Demo31</span><span class="p">.</span><span class="k">go</span><span class="p">:</span><span class="mi">16</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span> <span class="nx">inlining</span> <span class="nx">call</span> <span class="nx">to</span> <span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span>
</span></span><span class="line"><span class="cl"><span class="p">.</span><span class="o">/</span><span class="nx">Demo31</span><span class="p">.</span><span class="k">go</span><span class="p">:</span><span class="mi">6</span><span class="p">:</span><span class="mi">2</span><span class="p">:</span> <span class="nx">moved</span> <span class="nx">to</span> <span class="nx">heap</span><span class="p">:</span> <span class="nx">n</span>
</span></span><span class="line"><span class="cl"><span class="p">.</span><span class="o">/</span><span class="nx">Demo31</span><span class="p">.</span><span class="k">go</span><span class="p">:</span><span class="mi">7</span><span class="p">:</span><span class="mi">9</span><span class="p">:</span> <span class="kd">func</span> <span class="nx">literal</span> <span class="nx">escapes</span> <span class="nx">to</span> <span class="nx">heap</span>
</span></span><span class="line"><span class="cl"><span class="p">.</span><span class="o">/</span><span class="nx">Demo31</span><span class="p">.</span><span class="k">go</span><span class="p">:</span><span class="mi">14</span><span class="p">:</span><span class="mi">24</span><span class="p">:</span> <span class="kd">func</span> <span class="nx">literal</span> <span class="nx">does</span> <span class="nx">not</span> <span class="nx">escape</span>
</span></span><span class="line"><span class="cl"><span class="p">.</span><span class="o">/</span><span class="nx">Demo31</span><span class="p">.</span><span class="k">go</span><span class="p">:</span><span class="mi">15</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span> <span class="o">...</span> <span class="nx">argument</span> <span class="nx">does</span> <span class="nx">not</span> <span class="nx">escape</span>
</span></span><span class="line"><span class="cl"><span class="p">.</span><span class="o">/</span><span class="nx">Demo31</span><span class="p">.</span><span class="k">go</span><span class="p">:</span><span class="mi">15</span><span class="p">:</span><span class="mi">23</span><span class="p">:</span> <span class="err">~</span><span class="nx">R0</span> <span class="nx">escapes</span> <span class="nx">to</span> <span class="nx">heap</span>
</span></span><span class="line"><span class="cl"><span class="p">.</span><span class="o">/</span><span class="nx">Demo31</span><span class="p">.</span><span class="k">go</span><span class="p">:</span><span class="mi">16</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span> <span class="o">...</span> <span class="nx">argument</span> <span class="nx">does</span> <span class="nx">not</span> <span class="nx">escape</span>
</span></span><span class="line"><span class="cl"><span class="p">.</span><span class="o">/</span><span class="nx">Demo31</span><span class="p">.</span><span class="k">go</span><span class="p">:</span><span class="mi">16</span><span class="p">:</span><span class="mi">23</span><span class="p">:</span> <span class="err">~</span><span class="nx">R0</span> <span class="nx">escapes</span> <span class="nx">to</span> <span class="nx">heap</span>
</span></span></code></pre></td></tr></table>
</div>
</div><!-- raw HTML omitted -->
<!-- raw HTML omitted -->
<p>一般情况下,对于需要修改原对象值,或占用内存比较大的结构体,选择传指针。对于只读的占用内存较小的结构体,直接传值能够获得更好的性能</p>
<ul>
<li>
<p>参考资料</p>
<p><a class="link" href="https://geektutu.com/post/hpg-escape-analysis.html#3-%E5%A6%82%E4%BD%95%E5%88%A9%E7%94%A8%E9%80%83%E9%80%B8%E5%88%86%E6%9E%90%E6%8F%90%E5%8D%87%E6%80%A7%E8%83%BD" target="_blank" rel="noopener"
>Go 逃逸分析</a></p>
</li>
</ul>
</description>
</item>
<item>
<title>SQL优化</title>
<link>https://MoonerHigh.github.io/p/sql%E4%BC%98%E5%8C%96/</link>
<pubDate>Fri, 13 Jan 2023 10:48:35 +0800</pubDate>
<guid>https://MoonerHigh.github.io/p/sql%E4%BC%98%E5%8C%96/</guid>
<description><h1 id="sql优化">SQL优化</h1>
<ul>
<li>
<p>查询SQL尽量不要使用select *,而是使用具体字段</p>
<ol>
<li>select * 进行查询时,很可能不会用到索引,避免造成全表扫描</li>
<li>节省资源,减少网络开销</li>
</ol>
</li>
<li>
<p>避免在where子句中使用<code>or</code>来连接条件</p>
<ul>
<li>反例</li>
</ul>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">select</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">employee</span><span class="w"> </span><span class="k">where</span><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="k">or</span><span class="w"> </span><span class="n">salary</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">3000</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></td></tr></table>
</div>
</div><ul>
<li>正例</li>
</ul>
<p>使用union all</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">select</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">employee</span><span class="w"> </span><span class="k">where</span><span class="w"> </span><span class="n">id</span><span class="o">=</span><span class="mi">1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">union</span><span class="w"> </span><span class="k">all</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">select</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">employee</span><span class="w"> </span><span class="k">where</span><span class="w"> </span><span class="n">salary</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">3000</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></td></tr></table>
</div>
</div><p>分开两条SQL写</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">select</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">employee</span><span class="w"> </span><span class="k">where</span><span class="w"> </span><span class="n">id</span><span class="o">=</span><span class="mi">1</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">select</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">employee</span><span class="w"> </span><span class="k">where</span><span class="w"> </span><span class="n">salary</span><span class="o">=</span><span class="mi">3000</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></td></tr></table>
</div>
</div><!-- raw HTML omitted -->
<!-- raw HTML omitted -->
</li>
</ul>
</description>
</item>
<item>
<title>Archives</title>
<link>https://MoonerHigh.github.io/archives/</link>
<pubDate>Sun, 06 Mar 2022 00:00:00 +0000</pubDate>
<guid>https://MoonerHigh.github.io/archives/</guid>
<description></description>
</item>
<item>
<title>Evaluation</title>
<link>https://MoonerHigh.github.io/categories/evaluation/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://MoonerHigh.github.io/categories/evaluation/</guid>
<description></description>
</item>
<item>
<title>Links</title>
<link>https://MoonerHigh.github.io/links/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://MoonerHigh.github.io/links/</guid>
<description><p>To use this feature, add <code>links</code> section to frontmatter.</p>
<p>This page&rsquo;s frontmatter:</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span><span class="lnt">9
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">links</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">title</span><span class="p">:</span><span class="w"> </span><span class="l">GitHub</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="l">GitHub is the world&#39;s largest software development platform.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">website</span><span class="p">:</span><span class="w"> </span><span class="l">https://github.com</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">https://github.githubassets.com/images/modules/logos_page/GitHub-Mark.png</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">title</span><span class="p">:</span><span class="w"> </span><span class="l">TypeScript</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="l">TypeScript is a typed superset of JavaScript that compiles to plain JavaScript.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">website</span><span class="p">:</span><span class="w"> </span><span class="l">https://www.typescriptlang.org</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">ts-logo-128.jpg</span><span class="w">
</span></span></span></code></pre></td></tr></table>
</div>
</div><p><code>image</code> field accepts both local and external images.</p>
</description>
</item>
<item>
<title>Note</title>
<link>https://MoonerHigh.github.io/categories/note/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://MoonerHigh.github.io/categories/note/</guid>
<description></description>
</item>
<item>
<title>Search</title>
<link>https://MoonerHigh.github.io/search/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://MoonerHigh.github.io/search/</guid>
<description></description>
</item>
</channel>
</rss>