search.xml

<?xml version="1.0" encoding="utf-8"?>
<search> 
  
  
    
    <entry>
      <title>Patient Representation</title>
      <link href="/2024/03/19/patient-representation/"/>
      <url>/2024/03/19/patient-representation/</url>
      
        <content type="html"><![CDATA[<h1 id="Deep-representation-learning-of-patient-data-from-Electronic-Health-Records-EHR-A-systematic-review"><a href="#Deep-representation-learning-of-patient-data-from-Electronic-Health-Records-EHR-A-systematic-review" class="headerlink" title="Deep representation learning of patient data from Electronic Health Records (EHR): A systematic review"></a>Deep representation learning of patient data from Electronic Health Records (EHR): A systematic review</h1><h2 id="Abstract"><a href="#Abstract" class="headerlink" title="Abstract"></a>Abstract</h2><ul><li>Patient representation learning refers to learning a dense mathematical representation of a patient that encodes meaningful information from Electronic Health Records (EHRs).</li><li>The existing predictive models mainly focus on the prediction of single diseases, rather than considering the complex mechanisms of patients from a holistic review.</li><li>Advances in patient representation learning techniques will be essential for powering patient-level EHR analyses.</li></ul><h2 id="Introduction"><a href="#Introduction" class="headerlink" title="Introduction"></a>Introduction</h2><ul><li>In Electronic Health Records (EHRs), information regarding patient status is extensively documented. Therefore, EHR data provides a feasible mechanism to <strong>track patient health information</strong> and to make better decisions based on data-driven technologies.</li><li>Unlike data in clinical trials or other biomedical studies, secondary data extracted from EHRs are not designed to answer a specific hypothesis. Instead, their primary goal is to <strong>monitor a patient</strong>.</li><li>This results in the issue that EHR data have many challenging characteristics such as <strong>uncurated</strong> (data are not carefully chosen and thoughtfully organized or presented), <strong>poor-quality</strong> (data are rarely subject to data quality audits), <strong>high-dimensional</strong> (thousands of distinct medical events), <strong>sparse</strong> (lots of zero values), <strong>heterogeneous</strong> (drawn from different resources), <strong>temporal</strong> (data are collected over time), <strong>incomplete</strong> (missing values), <strong>large-scale</strong> (a large volume of data), and <strong>multimodal</strong> (multiple data modalities).</li><li>非精挑细选的、质量差、高维、稀疏、异构、时序、不完整、大规模、多模态</li></ul><h2 id="Background"><a href="#Background" class="headerlink" title="Background"></a>Background</h2><h3 id="Patient-learning-data-pipeline"><a href="#Patient-learning-data-pipeline" class="headerlink" title="Patient learning data pipeline"></a>Patient learning data pipeline</h3><ul><li>Pipeline: raw data -&gt; embedding technique -&gt; patient representation</li><li>Learning strategies: unsupervised, supervised, or self-supervised.</li><li>Evaluate: <ul><li>Clinical tasks: mortality, readmission, or a specific disease prediction.</li><li>Visualization: intuitive understanding or interpretation.</li></ul></li></ul><h3 id="Patient-representation-methods"><a href="#Patient-representation-methods" class="headerlink" title="Patient representation methods"></a>Patient representation methods</h3><ul><li>Vector-based:<ul><li>Fully connected deep neural network (DNN): Multi-layer perceptrons (MLPs)<ul><li><a href="https://dl.acm.org/doi/abs/10.1145/2783258.2783365" target="_blank" rel="noopener">Deep computational phenotyping</a>:</li><li><a href=""> Scalable and accurate deep learning with electronic health records</a></li></ul></li></ul></li></ul>]]></content>
      
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
            <tag> Medicine </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>NLP models for clinical</title>
      <link href="/2023/10/26/llm4clinical/"/>
      <url>/2023/10/26/llm4clinical/</url>
      
        <content type="html"><![CDATA[<h1 id="Transformer-architectures"><a href="#Transformer-architectures" class="headerlink" title="Transformer architectures"></a>Transformer architectures</h1><ul><li>Text:<ul><li>BioBERT[1]: a domain-specific language representation model pre-trained on large-scale biomedical corpora.</li><li>PubMedBERT[2]: biomedical literature from PubMed.</li><li>BioMegatron[3]: PubMed-derived free text.</li><li>ClinicalBERT[4]: clinical text in MIMIC-III.</li><li>GatorTron[5]: using &gt;90 billion words of text from the de-identified clinical notes of University of Florida (UF) Health, PubMed articles, and Wikipedia.</li></ul></li><li>EHR:</li></ul><p>[1] Lee, J. et al. BioBERT: a pre-trained biomedical language representation model for<br>biomedical text mining. Bioinformatics. 36, 1234–1240 (2020).<br>[2] Gu, Y. et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans. Comput. Healthc. 3, 1–23 (2022).<br>[3] Shin, H.-C. et al. BioMegatron: larger biomedical domain language model. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 4700–4706 (2020).<br>[4] Alsentzer, E. et al. Publicly Available Clinical BERT Embeddings. in Proc. 2nd Clinical Natural Language Processing Workshop 72–78 (2019).<br>[5] Yang X, Chen A, PourNejatian N, et al. A large language model for electronic health records[J]. NPJ Digital Medicine, 2022, 5(1): 194.</p>]]></content>
      
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
            <tag> Medicine </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Latex</title>
      <link href="/2023/08/02/latex/"/>
      <url>/2023/08/02/latex/</url>
      
        <content type="html"><![CDATA[<p>记录一些关于Latex的语法。</p><ul><li><p><a href="http://mohu.org/info/symbols/symbols.htm" target="_blank" rel="noopener">符号</a></p></li><li><p><a href="https://blog.csdn.net/weixin_41519463/article/details/103737464" target="_blank" rel="noopener">表格</a><br>主要是<code>multirow</code>和<code>multicolumn</code>的使用。</p><ul><li><code>multirow</code>需要引入相关包：<br><code>\usepackage{multirow}</code></li><li>格式：<br>行： <code>\multirow{NUMBER_OF_ROWS}{WIDTH}{CONTENT}</code><br>列：<code>\multicolumn{NUMBER_OF_COLUMNS}{ALIGNMENT}{CONTENT}</code></li><li>示例：<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><span class="line">\usepackage&#123;multirow&#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">\begin&#123;table*&#125;[t]</span><br><span class="line">\centering</span><br><span class="line">\caption&#123;MAP scores of teacher model, different student models with 4 widths and three baseline models with different length of binary codes on CIFAR-10 and SUN datasets.&#125;</span><br><span class="line">\label&#123;table1&#125;</span><br><span class="line">\begin&#123;tabular&#125;&#123;|c|c|c|c|c|c|c|c|c|c|c|c|&#125;</span><br><span class="line">\hline</span><br><span class="line">\multicolumn&#123;2&#125;&#123;|c|&#125;&#123;\multirow&#123;2&#125;&#123;*&#125;&#123;Model&#125;&#125; &amp;\multirow&#123;2&#125;&#123;*&#125;&#123;FLOPs&#125;&amp;\multirow&#123;2&#125;&#123;*&#125;&#123;Params&#125; &amp; \multicolumn&#123;4&#125;&#123;c|&#125;&#123;CIFAR-10&#125;&amp;\multicolumn&#123;4&#125;&#123;c|&#125;&#123;SUN&#125;\\</span><br><span class="line">\cline&#123;5-12&#125;</span><br><span class="line"></span><br><span class="line">\multicolumn&#123;2&#125;&#123;|c|&#125;&#123;&#125; &amp; &amp; &amp; 12bits &amp; 24bits &amp; 32bits &amp; 48bits &amp; 12bits &amp; 24bits &amp; 32bits &amp; 48bits \\</span><br><span class="line">\hline</span><br><span class="line"></span><br><span class="line">\multicolumn&#123;2&#125;&#123;|c|&#125;&#123;Teacher&#125; &amp;4.12G    &amp;25.56M &amp;0.87841    &amp;0.89512    &amp;0.9014 &amp;0.90601    &amp;0.83587    &amp;0.85736    &amp;0.86297    &amp;0.87103\\ </span><br><span class="line">\hline</span><br><span class="line"></span><br><span class="line">%0.25x-----------------</span><br><span class="line">\multirow&#123;4&#125;&#123;*&#125;&#123;$0.25\times$&#125; &amp; Stu-1 &amp; 0.15G &amp; 1.03M &amp;</span><br><span class="line">0.70746 &amp; 0.73458 &amp; 0.74909 &amp;   0.75833 &amp;   0.69618 &amp;   0.76631 &amp;   0.78075 &amp;   0.78787 \\</span><br><span class="line">\cline&#123;2-12&#125;</span><br><span class="line"></span><br><span class="line">\multirow&#123;4&#125;&#123;*&#125;&#123;&#125; &amp; Stu-2 &amp;0.19G    &amp;1.08M  &amp;0.7629 &amp;0.79111    &amp;0.80039    &amp;0.80519    &amp;0.73539    &amp;0.79714    &amp;0.80753    &amp;0.81195\\</span><br><span class="line">\cline&#123;2-12&#125;</span><br><span class="line"></span><br><span class="line">\multirow&#123;4&#125;&#123;*&#125;&#123;&#125; &amp; Stu-3 &amp;0.26G    &amp;1.43M  &amp;0.84684    &amp;0.86443    &amp;0.87384    &amp;0.88268    &amp;0.79284    &amp;0.83442    &amp;0.84350    &amp;0.84353\\</span><br><span class="line">\cline&#123;2-12&#125;</span><br><span class="line"></span><br><span class="line">\multirow&#123;4&#125;&#123;*&#125;&#123;&#125; &amp; Stu-4 &amp; 0.29G   &amp;1.99M  &amp;0.85901    &amp;0.87269    &amp;0.8836 &amp;0.88728    &amp;0.81997    &amp;0.84620    &amp;0.85041    &amp;0.85036\\</span><br><span class="line">\hline</span><br><span class="line">\end&#123;tabular&#125;</span><br><span class="line">\label&#123;table_MAP&#125;</span><br><span class="line">\end&#123;table*&#125;</span><br></pre></td></tr></table></figure></li></ul></li></ul>]]></content>
      
      
      <categories>
          
          <category> Notes </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Latex </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>ML for Emergency</title>
      <link href="/2023/04/06/emergency-ML/"/>
      <url>/2023/04/06/emergency-ML/</url>
      
        <content type="html"><![CDATA[<h1 id="综述"><a href="#综述" class="headerlink" title="综述"></a>综述</h1><h2 id="Critical-Care"><a href="#Critical-Care" class="headerlink" title="Critical Care"></a>Critical Care</h2><blockquote><p>Kang D-Y, Cho K-J, Kwon O, et al. <strong>Artificial intelligence algorithm to predict the need for critical care in prehospital emergency medical services</strong>. Scandinavian journal of trauma,  resuscitation and emergency medicine. 2020;28:1-8. </p></blockquote><h2 id="Triage"><a href="#Triage" class="headerlink" title="Triage"></a>Triage</h2><blockquote><p>Sánchez-Salmerón R, Gómez-Urquiza JL, Albendín-García L, et al. <strong>Machine learning methods applied to triage in emergency services: A systematic review</strong>. International Emergency Nursing. 2022;60:101109, </p></blockquote>]]></content>
      
      
      <categories>
          
          <category> AI for Medicine </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
            <tag> Medicine </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>急危重症预测</title>
      <link href="/2023/03/22/acute/"/>
      <url>/2023/03/22/acute/</url>
      
        <content type="html"><![CDATA[<h2 id="急诊预检分诊专家共识-1"><a href="#急诊预检分诊专家共识-1" class="headerlink" title="急诊预检分诊专家共识[1]"></a>急诊预检分诊专家共识[1]</h2><h3 id="相关概念"><a href="#相关概念" class="headerlink" title="相关概念"></a>相关概念</h3><ul><li>急诊预检分诊：对急诊患者快速评估，根据急危重程度进行优先顺序的分级与分流。</li><li>分级标准：客观指标，人工评级指标，设定相应时限和分级预警标识。</li><li>响应时限：最长等候时间</li><li>候诊时间：实际等候就诊时间</li></ul><h3 id="原则"><a href="#原则" class="headerlink" title="原则"></a>原则</h3><ul><li>急危重症优先就诊：抓住威胁患者生命的主要矛盾，分清轻重缓急，遵循从重到轻、从病情迅速变化到相对稳定的就诊原则。</li><li>准确快速分级分区：依据科学标准进行准确分级，安排有能力的分诊人员和借助敏感性高的分诊工具进行快速、准确分诊，并合理建设规划急诊患者的就诊区域，实现分区救治，提高急诊患者就诊的安全性及高效性。</li><li>动态评估及时预警：预判患者的病情及潜在的危险，采取相应的医疗护理措施予以动态评估，比如设定可控的最短响应时限、危重患者实时检测生命体征、设立巡回评估岗位等，及时发现病情变化、识别影响临床结局的紧急程度指标和实现及时预警的效果。</li><li>以人为本有效沟通</li></ul><h3 id="预检分诊级别"><a href="#预检分诊级别" class="headerlink" title="预检分诊级别"></a>预检分诊级别</h3><ul><li>I级：急危患者，需要立即得到救治。正在或即将发生生命威胁或病情恶化。</li><li>II级：急重患者，评估与救治同时进行。病情危重或迅速恶化，如不能进行即刻治疗则危及生命危险或造成严重器官功能衰竭，或短时间内进行治疗对预后产生重大影响。</li><li>III级：急症患者，短时间内得到救治。存在潜在的生命危险，如短时间内不进行干预，病情可能进展至威胁生命或产生十分不利的后果。</li><li>IV级：亚急症或非急症患者。亚急症存在潜在的严重性，一段时间内未予治疗，患者情况可能会恶化或出现不利结局，或症状加重及持续时间延长；非急症患者具有慢性或非常轻微的症状，即便等待较长时间再进行治疗也不会有太大影响。</li></ul><h3 id="级别评定标准"><a href="#级别评定标准" class="headerlink" title="级别评定标准"></a>级别评定标准</h3><ul><li>标准[1]：<ul><li>客观评估指标：患者生命体征、即使检验与检查，包括心率、呼吸、血压、氧合、心电图、血糖、心肌酶等。</li><li>人工评级标准：将患者的症状合体征按疾病严重程度进行划分。主要症状体征：气道、呼吸、循环、意识为主。</li></ul></li><li>标准[2]：<ul><li>单项客观指标：脉搏、呼吸频率、收缩压、经皮血氧饱和度（<script type="math/tex">SpO_2</script>）、体温</li><li>综合指标：改良早期预警评分：呼吸、体温、收缩压、心率、AVPU反应</li><li>症状/体征指标：心搏/呼吸骤停、气道阻塞/窒息、休克征象、突发意识丧失、急性中毒危及生命等。</li></ul></li></ul><h3 id="响应时限"><a href="#响应时限" class="headerlink" title="响应时限"></a>响应时限</h3><ul><li>I级：即刻</li><li>II级：10min</li><li>III级：30min</li><li>IV级：亚急症 60min，非急症2~4h</li></ul><h3 id="分区管理"><a href="#分区管理" class="headerlink" title="分区管理"></a>分区管理</h3><ul><li>I级：复苏区，即刻抢救</li><li>II级：抢救区</li><li>III级：优先诊疗区</li><li>IV级：普通诊疗区</li></ul><h3 id="就诊流程"><a href="#就诊流程" class="headerlink" title="就诊流程"></a>就诊流程</h3><ul><li>充分评估：<ul><li>重点询问和评估，包括气道、呼吸、循环、意识等方面</li><li>迅速掌握主要症状、主诉、生命体征等</li><li>分诊思路，从重症到轻症，将致命性疾病放在首位。</li><li>评估内容：患者基本资料、来院方式、客观指标、主诉、症状、体征、目前主要问题（临床表现）、相关病史、检验结果、初始分诊级别（再次分诊时间和原因）、评估和诊疗区域、候诊时间、可能采取的抢救或治疗措施等。</li></ul></li><li>准确定级</li><li>有效沟通</li><li>妥善接诊</li><li>动态评估：预检评估，确保患者在响应时限内得到安全救治；设置再评估时间，等候时间超过响应时限，立即启动再次评估，重新确认就诊级别；患者在候诊过程重出现病情变化，或获得了影响患者紧急程度的新信息，需要重新分诊并及时调整就诊级别。</li></ul><h3 id="参考文献"><a href="#参考文献" class="headerlink" title="参考文献"></a>参考文献</h3><p>[1] 史冬雷, 刘晓颖, 周瑛. 急诊预检分诊专家共识[J]. 中华急诊医学杂志, 2018, 27(6): 599-604.<br>[2] 中华护理学会急诊专业委员会, 浙江省急诊医学质量控制中心. 急诊预检分级分诊标准[J]. 中华急诊医学杂志, 2016, 25(4): 415-417.</p>]]></content>
      
      
      <categories>
          
          <category> Notes </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Medicine </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Advances in Mining Heterogeneous Healthcare Data</title>
      <link href="/2023/03/09/heterogeneousHealthcare/"/>
      <url>/2023/03/09/heterogeneousHealthcare/</url>
      
        <content type="html"><![CDATA[<p><a href="https://dl.acm.org/doi/10.1145/3447548.3470789?sid=SCITRUS" target="_blank" rel="noopener">KDD’21 tutorial</a></p><p>EHR数据：</p><ul><li>结构化:<ul><li>人口统计资料（demographics)：年龄、性别、经济状况、保险类型、语言、种族、居住条件、家庭结构、地理位置、工作</li><li>用药（medications）：医药处方、非处方药、毒品、酒精<ul><li>代码：NDC, RxNorm</li></ul></li><li>实验室结果（laboratory results）：血液、尿液、粪便、唾液、脊髓液、腹水、混合液、骨髓、肺<ul><li>代码：LOINC</li></ul></li><li>账单（billing）：<ul><li>诊断（diagnoses）:ICD-9,10</li><li>程序（procedures）：CPT, ICD</li></ul></li><li>持续的监护数据（continuous monitoring data）：心率、脉搏、呼吸率、体温、氧饱和度、血压</li><li>多种类型的医学代码（multi-typed medical codes）</li></ul></li><li>非结构化：<ul><li>临床笔记与报告（clinical notes and reports）：出院小结，主治与住院，护士，专科（放射、病理、心电图、营养、呼吸），会诊医生、咨询医生、急诊科</li><li>医学图像（medical images）：X-ray, CT, MRI, PET</li></ul></li></ul><p>任务（analytics tasks）：</p><ul><li>表型（phenotyping）：心律不齐、肾衰竭、失眠</li><li>风险预测</li><li>用药建议</li><li>疾病分类</li><li>医学语言翻译</li><li>医学报告生成</li><li>临床试验挖掘</li></ul><p>结构化健康数据（structured health data）</p><ul><li>表型：从EHR数据中学习医学概念表示<ul><li>Med2Vec, dipole, GRAM, KAME, HAP, Integrating multimodal EHR,</li></ul></li><li>疾病/风险预测<ul><li>RETAIN, LSAN, HAM, TAM, HiTANet, MedPath, MedRetriever</li></ul></li><li>治疗建议<ul><li>LEAP, GAMENet</li></ul></li></ul><p>非结构化数据</p><ul><li>自动ICD编码/疾病分类<ul><li>HyperCore, C-MemNN, CAML, Fusion</li></ul></li><li>医学语言翻译</li><li>医学报告生成</li><li>临床试验挖掘</li></ul>]]></content>
      
      
      <categories>
          
          <category> AI for Medicine </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
            <tag> Medicine </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Association mining</title>
      <link href="/2023/02/28/association/"/>
      <url>/2023/02/28/association/</url>
      
        <content type="html"><![CDATA[<h2 id="数据挖掘"><a href="#数据挖掘" class="headerlink" title="数据挖掘"></a><a href="https://www.51cto.com/article/698009.html" target="_blank" rel="noopener">数据挖掘</a></h2><p>数据挖掘(data mining)又译为资料探勘、数据采矿，是指从大量的、不完全的、有噪声的、模糊的、随机的数据中提取隐含在其中的、人们事先不知道的但又潜在有用的信息和知识的过程。</p>]]></content>
      
      
      <categories>
          
          <category> Machine Learning </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>迁移强化学习</title>
      <link href="/2023/02/17/transRL/"/>
      <url>/2023/02/17/transRL/</url>
      
        <content type="html"><![CDATA[<p>参考文献：<a href="https://www.zhihu.com/column/c_1484160649094131712" target="_blank" rel="noopener">迁移强化学习</a>, <a href="https://cloud.tencent.com/developer/article/1541899" target="_blank" rel="noopener">迁移学习在强化学习中的应用及进展</a></p><h2 id="相同状态动作空间下的迁移-the-same-state-observation-action-transfer"><a href="#相同状态动作空间下的迁移-the-same-state-observation-action-transfer" class="headerlink" title="相同状态动作空间下的迁移(the same state(observation)-action transfer)"></a>相同状态动作空间下的迁移(the same state(observation)-action transfer)</h2><ul><li>迁移内容：<ul><li>具体知识(specific knowledge): <ul><li>策略(policy)：<ul><li>直接更新(reuse for update):直接使用知识对策略或者值函数进行更新，如策略蒸馏</li><li>辅助探索(reuse for exploration)：使用源域上的知识帮助目标域上的智能体去探索，如策略复用</li></ul></li><li>价值函数(value function)</li><li>示范(demonstrations)</li></ul></li><li>抽象知识(abstract knowledge)</li></ul></li></ul>]]></content>
      
      
      <categories>
          
          <category> Machine Learning </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>MIMIC 数据集</title>
      <link href="/2023/02/16/mimic/"/>
      <url>/2023/02/16/mimic/</url>
      
        <content type="html"><![CDATA[<p>参考资料：<a href="https://mimic.mit.edu/docs/" target="_blank" rel="noopener">MIMIC文档</a>，<a href="https://physionet.org/content/?topic=MIMIC" target="_blank" rel="noopener">MIMIC数据集</a>，<a href="https://blog.csdn.net/qq_43787862/article/details/105028846" target="_blank" rel="noopener">MIMIC-III数据集介绍</a></p><h2 id="MIMIC-III"><a href="#MIMIC-III" class="headerlink" title="MIMIC-III"></a><a href="https://physionet.org/content/mimiciii/1.4/" target="_blank" rel="noopener">MIMIC-III</a></h2><ul><li>患者人口统计学信息与院内周转信息<ul><li><code>PATIENTS</code>: 患者信息<ul><li>SUBJECT_ID, GENDER </li><li>DOB, DOD, DOD_HOSP, DOD_SSN</li></ul></li><li><code>ADMISSIONS</code>:患者入院信息<ul><li>SUBJECT_ID, HADM_ID: patients id, admission id</li><li>ADMITTIME, DISCHTIME, DEATHTIME</li><li>ADMISSION_TYPE, ADMISSION_LOCATION</li><li>INSURANCE, LANGUAGE, RELIGION, MARITAL_STATUS, ETHNICITY </li><li>EDREGTIME, EDOUTTIME: emergency</li><li>DIAGNOSIS: preliminary, free text diagnosis</li><li>HOSPITAL_EXPIRE_FLAG: dead in hospital </li></ul></li><li><code>ICUSTAYS</code>: ICU入住信息<ul><li>SUBJECT_ID, HADM_ID, ICUSTAY_ID</li><li>FIRST_CAREUNIT, LAST_CAREUNIT, FIRST_WARDID, LAST_WARDID</li><li>INTIME, OUTTIME, LOS</li></ul></li><li><code>CALLOUT</code>: 患者离开ICU信息</li><li><code>SERVICES</code>: 医疗服务</li><li><code>TRANSFERS</code>: 周转信息</li></ul></li><li>患者在监护室住院期间采集的信息<ul><li><code>CAREGIVERS</code>: 看护者信息</li><li><code>CHATEVENTS</code>:电子图表，生命体征，护理相关的附加信息:呼吸机设置、实验室值、代码状态、精神状态等等。<ul><li>SUBJECT_ID, HADM_ID, ICUSTAY_ID</li><li>ITEMID</li><li>CHARTTIME, STORETIME</li><li>CGID</li><li>VALUE, VALUENUM</li></ul></li><li><code>DATETIMEEVENTS</code>: 操作信息</li><li><code>INPUTEVENTS_CV</code>: 入量信息</li><li><code>INPUTEVENTS_MV</code>:</li><li><code>NOTEEVENTS</code>: 治疗记录</li><li><code>OUTPUTEVENTS</code>: 出量信息</li><li><code>PROCEDUREEVENTS_MV</code>: 操作或手术信息</li></ul></li><li>医院记录系统采集的信息   <ul><li><code>CPTEVENTS</code>: 现行程序术语(current procedural terminology, CPT)<ul><li>SUBJECT_ID, HADM_ID</li><li>COSTCENTER: 支付中心，<code>ICU</code>支付或<code>Resp</code>对应于机械或无创通气，由呼吸治疗师收费</li><li>CHARTDATE</li><li>CPT_CD, CPT_NUMBER, CPT_SUFFIX</li><li>TICKET_ID_SEQ</li><li>SECTIONHEADER, SUBSECTIONHEADER: CPT代码分类</li></ul></li><li><code>DIAGNOSES_ICD</code>: ICD diagnoses<ul><li>SUBJECT_ID, HADM_ID</li><li>SEQ_NUM, ICD9_CODE</li></ul></li><li><code>DRGCODES</code>: 诊断类别组</li><li><code>LABEVENTS</code>: 化验项目</li><li><code>MICROBIOLOGYEVENT</code>: 微生物病原体检测</li><li><code>PRESCRIPTIONS</code>: 用药记录</li><li><code>PROCEDURES_ICD</code>: 手术记录</li></ul></li><li>字典信息<ul><li><code>D_CPT.csv</code>: 操作记录</li><li><code>D_ICD_DIAGNOSES.csv</code>：诊断</li><li><code>D_ICD_PROCEDURES.csv</code>: 手术代码</li><li><code>D_ITEMS.csv</code>：记录项目</li><li><code>D_LABITEMS</code>: 化验项目</li></ul></li></ul><h2 id="MIMIC-III-Waveform"><a href="#MIMIC-III-Waveform" class="headerlink" title="MIMIC-III Waveform"></a><a href="https://physionet.org/content/mimic3wdb/1.0/" target="_blank" rel="noopener">MIMIC-III Waveform</a></h2><p><a href="https://physionet.org/content/mimic3wdb-matched/1.0/" target="_blank" rel="noopener">Subset</a></p><h2 id="MIMIC-IV"><a href="#MIMIC-IV" class="headerlink" title="MIMIC-IV"></a><a href="https://physionet.org/content/mimiciv/2.2/" target="_blank" rel="noopener">MIMIC-IV</a></h2><h2 id="MIMIC-IV-ED"><a href="#MIMIC-IV-ED" class="headerlink" title="MIMIC-IV-ED"></a><a href="https://physionet.org/content/mimic-iv-ed/2.2/" target="_blank" rel="noopener">MIMIC-IV-ED</a></h2><h2 id="MIMIC-CXR"><a href="#MIMIC-CXR" class="headerlink" title="MIMIC-CXR"></a><a href="https://physionet.org/content/mimic-cxr/2.0.0/" target="_blank" rel="noopener">MIMIC-CXR</a></h2>]]></content>
      
      
      <categories>
          
          <category> Notes </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Medicine </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>医疗保障急救技术</title>
      <link href="/2023/02/03/emergency/"/>
      <url>/2023/02/03/emergency/</url>
      
        <content type="html"><![CDATA[<p>参考：<a href="https://book.sciencereading.cn/shop/book/Booksimple/show.do?id=BC4ED798F85D16904E053010B0A0AAD86000" target="_blank" rel="noopener">大型活动医疗保障急救技术指导手册</a></p><h2 id="中国急救（China-Emergency-Medical-Services）"><a href="#中国急救（China-Emergency-Medical-Services）" class="headerlink" title="中国急救（China Emergency Medical Services）"></a>中国急救（China Emergency Medical Services）</h2><p>降低突发疾病和创伤等意外伤害的早期死亡率，提高救治成功率，降低后期伤残率的关键在于早期救治的规范性、及时性。</p><ol><li><p>医疗卫生保障工作的目标与内容</p><ul><li>工作目标：<ul><li>保障大型活动顺利进行</li><li>维护参与者的身体健康，避免或减少活动期间的人员伤亡</li><li>帮助大型活动组织者解决与生命健康相关的运行风险和危机</li></ul></li><li>工作内容：<ul><li>现场医疗急救</li><li>慢性疾病的日常处理</li><li>突发事件应急响应</li><li>传染病防控</li><li>食品卫生管理</li><li>兴奋剂检测</li></ul></li><li>工作原则<ul><li>建立统一高效的指挥体系和工作机制</li><li>执行国际化医疗卫生服务标准</li><li>坚持项目保障和城市运行两手抓</li></ul></li></ul></li><li><p>基本要求</p><ul><li>服务时间、地点及项目内容：确定工作时可能会遇到的天气状况、地理特点和服务项目特征。</li><li>项目内容：根据项目特点，制定针对性方案。</li><li>服务对象：确认参与人员的数量、年龄层次、身份职业、所处环境、聚散流向、民族习惯、主观意愿、特殊需求等。</li><li>工作内容：<ul><li>场馆内：现场危重病抢救、紧急监护转运、慢性疾病管理与日常诊疗、大型公共卫生事件应急响应、兴奋剂检测等。</li><li>城市服务：在定点医院提供门诊、急诊、住院服务、传染病防控、灾难应急响应等。</li></ul></li><li>工作目标：<ul><li>08年奥运会：“三个满意”，“三化”，“三为”，“三无”</li><li>东奥会：快速救援、精准保障、专业操作、平稳安全、沟通顺畅</li></ul></li></ul></li><li><p>医疗卫生保障功能单位设置</p><ul><li>基本设置：<ul><li>场边医疗急救工作点(field of play, FOP)：比赛现场医疗工作点、观众医疗工作点、重点贵宾医疗工作点。</li><li>场馆医疗站：专业医疗抢救为主、日常医疗处置为辅。根据服务人群类别和服务场地进行设置，如运动员医疗站与观众医疗站。</li><li>急救车系统：每个医疗站1~2车。急救车均按照MICU（move intensive care unit，移动重症监护病房）标准配置。</li><li>综合诊所：为参与人员提供集中、全面医疗服务的综合性医疗机构。提供优质的服务，减轻对城市医疗服务资源的占用和影响。</li><li>城市医疗卫生服务：<ul><li>定点医院服务：制定保障方案和应急预案；专人负责，绿色通道；及时报告医疗救治信息。</li><li>公共卫生及血液保障<ul><li>疾病预防控制工作：“综合协调、专业负责、属地管理”，提升公共卫生安全保障的功能与水平，降低大型活动公共卫生事件发生概率。建立传染病检测网络，针对就诊人员开展发热、腹泻、黄疸、皮疹、结膜红肿、呕吐等症状监测。</li><li>血液保障：血液采集与储备，血液预警分级，紧急送血运输，应急相容性输血，运行保障演练。</li></ul></li></ul></li></ul></li></ul></li><li><p>现场急救</p><ul><li>现场救援流程：<ul><li>预警与报警，应急响应，现场处置，现场撤离，伤员后送，定点医院接诊。</li></ul></li><li>就医控制：<ul><li>三级处置：现场处置，医疗站处置，定点医院处置</li><li>二次分流：场馆医疗卫生保障团队首诊按病情分诊，城市定点医院首诊按需求分流。</li></ul></li><li>场馆医疗急救信息报送：<ul><li>800M数字集群通讯系统、移动电话系统、程控电话系统、因特网系统和书面文件。</li><li>时间节点、统一渠道、统一内容、统一汇总、统一出口。</li><li>医疗信息统计表系统</li></ul></li></ul></li><li><p>风险管理</p><ul><li>医疗运行风险：救援延误、信息中断、备援力量无法启动、关键设备故障、急救药品严重不足、医疗差错、救援通道堵塞、指挥系统瘫痪、救援人员重伤或殉职等。<ul><li>突发性、破坏性、不确定性、紧迫性、信息资源紧缺性、舆论关注性。</li></ul></li><li>危机管理：<ul><li>风险预防：问题（issue），突发事件（incident），危机（crisis）<ul><li>风险事件清单，针对性响应预案。</li></ul></li><li>风险监测：<ul><li>经验监测：一线人员对救援工作进展的现状和后果进行判断，对可能出现风险的潜在因素进行挖掘并汇报。</li><li>信息检测：指挥系统通过应急通信平台对医疗运行的各级部门的工作信息及意外事件进行及时的收集和分析。</li></ul></li><li>风险预警：量化风险指标，设立风险警戒线</li><li>危机处理：级别确认并启动预案；处理好突发情况；在最低层面充分解决问题；做好信息上报和事件跟踪。</li><li>大型特发事件应急响应：建立以场馆团队为基础，以城市应急医疗队伍为保障，以检伤分类卡为依据，以场馆周边指定医院为依托的突发事件处理原则和程序。</li></ul></li></ul></li></ol>]]></content>
      
      
      <categories>
          
          <category> Notes </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Medicine </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>脓毒症</title>
      <link href="/2023/01/12/sepsis/"/>
      <url>/2023/01/12/sepsis/</url>
      
        <content type="html"><![CDATA[<p>参考：《实用重症医学(第二版)》</p><h2 id="机体反应与器官功能不全"><a href="#机体反应与器官功能不全" class="headerlink" title="机体反应与器官功能不全"></a>机体反应与器官功能不全</h2><p>研究证明，机体对病损打击后产生的炎症反应的过程是影响患者病情的重要因素。机体反应是指感染、创伤、缺血等多种原因引发的机体对损伤的炎症反应。</p><h2 id="相关研究"><a href="#相关研究" class="headerlink" title="相关研究"></a>相关研究</h2><ul><li><a href="https://link.springer.com/article/10.1007/s00134-019-05872-y" target="_blank" rel="noopener">Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy</a></li></ul>]]></content>
      
      
      <categories>
          
          <category> AI for Medicine </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
            <tag> Medicine </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Prediction of Treatment Medicines</title>
      <link href="/2023/01/09/treatment-medicines-prediction/"/>
      <url>/2023/01/09/treatment-medicines-prediction/</url>
      
        <content type="html"><![CDATA[<p>Prediction of treatment medicines can assist doctors in making informed prescription decisions for patients according to their EHRs.</p><p>Predicting treatment medicines can provide insights for treatment planning and prescription management, thus improving the efficiency of clinical decision making and medical risk management. </p><h3 id="Challenge"><a href="#Challenge" class="headerlink" title="Challenge:"></a>Challenge:</h3><ul><li>异质性：heterogenous nature of EHR data that typically includes laboratory results, treatment records, disease conditions and demographic information. <ul><li>multiple sources and in different formats.</li><li>longitudinal information (sequences of laboratory examinations and treatment medicines) and static information (patient demographics).</li></ul></li><li>相关性：complex corrrelations among EHR sequences, including inter-correlations between sequences and temporal intra-correlations within each esquences.<ul><li>examination results, treatment prescriptions(retrospectively review).</li><li>laboratory results and treatment medicines are mutually informative.</li></ul></li><li>时间动态：temporal dynamics of these correlations changing with disease progression.<ul><li>food and drinks, mood.</li></ul></li></ul><h3 id="Related-work"><a href="#Related-work" class="headerlink" title="Related work"></a>Related work</h3><ul><li>homogeneous data(同质数据): T-LSTM[4], PacRNN[5], ATTAIN[6] (future diagnoses or recommend medications)</li><li>heterogeneouts events with multi-scale sampling(多尺度采样的异构事件)：HE-LSTM</li><li>multi-view sequential learning &amp; encoding heterogeneous sequences: GAMENet[9], DMNC[10]</li><li>correlations of heterogeneous sequences: LSTM-DE[11]</li></ul><h3 id="Weakness"><a href="#Weakness" class="headerlink" title="Weakness"></a>Weakness</h3><ul><li>ignore the sequential interactions in temporal sequences</li><li>fail to fully exploit past records for improving prediction performance.</li></ul><h2 id="Prediction-of-Treatment-Medicines-With-Dual-Adaptive-Sequential-Networks"><a href="#Prediction-of-Treatment-Medicines-With-Dual-Adaptive-Sequential-Networks" class="headerlink" title="Prediction of Treatment Medicines With Dual Adaptive Sequential Networks"></a><a href="https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&amp;arnumber=9328483" target="_blank" rel="noopener">Prediction of Treatment Medicines With Dual Adaptive Sequential Networks</a></h2><h3 id="Contributions"><a href="#Contributions" class="headerlink" title="Contributions"></a>Contributions</h3><ul><li>data heterogeneity &amp; time-varying correlations in logitudinal data:<ul><li>dual adaptive sequential networks(DASNet)</li><li>two decomposed adaptive LSTM(DA-LSTM): capture the inter-correlations ofheterogeneous sequences through a decomposition structure with auxiliary input.</li><li>an attentive meta learning network(AT-MetaNet): generates parameter weights in DA-LSTM, extract meta knowleged from historical sequences of examination results and treatment medicines.</li><li>an attentive fusion network (AF-FuNet): fuse the embedding representations of heterogeneous information sources</li></ul></li></ul>]]></content>
      
      
      <categories>
          
          <category> AI for Medicine </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
            <tag> Medicine </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Label Enhancement</title>
      <link href="/2022/12/15/label-enhancement/"/>
      <url>/2022/12/15/label-enhancement/</url>
      
        <content type="html"><![CDATA[<p>标记增强将逻辑标记转化为标记分布。相比于逻辑标记空间，标记分布空间包含了更多监督信息，如示例在空间中的拓扑关系、标记在空间中的相关性等。</p><h2 id="以往工作"><a href="#以往工作" class="headerlink" title="以往工作"></a>以往工作</h2><ul><li>依靠对数据的先验知识： 年龄估计中，假设年龄标记分布以高斯分布。</li><li>基于传统的模糊方法：基于C均值模糊聚类算法FCM，模糊SVM算法。</li><li>基于图模型：标记传播[1]，流形学习[2]。</li></ul><p>对于多义性对象而言，标记分布比逻辑标记更接近其监督信息的本质。描述度表示该标记对示例的描述程度在所有类别标记里所占比例。</p><ul><li>区别于概率标记[3]，概率标记在无法直接确定该对象的正确标记的情况下，用概率进行描述。</li><li>区别于模糊分类中的隶属度[4]，隶属度反应的是部分真实的状态，其变化范围为完全的正确到完全错误。</li></ul><h2 id="学习方法"><a href="#学习方法" class="headerlink" title="学习方法"></a>学习方法</h2><ul><li>基于先验知识：通过对数据的了解，知道标记分布满足的参数模型，如年龄估计[5]中，假设标记分布服从高斯分布，采用自适应方法确定先验分布中的参数，即标准差。</li><li>基于模糊方法：利用模糊数学的思想，通过模糊聚类、模糊运算和核隶属度等方法，挖掘标记间相关信息，将逻辑标记转化为标记分布。<ul><li>模糊C-均值聚类（Fuzzy C-Means Algorithm, FCM）</li><li>核隶属度（模糊支持向量机）</li></ul></li><li>基于图的标记增强</li></ul><p>参考文献：<br>[1] Yu-Kun Li, Min-Ling Zhang, and Xin Geng. Leveraging implicit relative labelingimportance information for effective multi-label learning. In Proceedings of the 15th IEEE International Conference on Data Mining, pages 251–260, Atlantic City, NJ, 2015.<br>[2] Peng Hou, Xin Geng, and Min-Ling Zhang. Multi-label manifold learning. In Proceedings of the 30th AAAI Conference on Artificial Intelligence, pages 1680–1686, Phoenix, AZ, 2016.<br>[3] Benjamin Quost and Thierry Denœux. Learning from data with uncertain labels by boosting credal classifiers. In Proceedings of the 1st ACM SIGKDD Workshop on Knowledge Discovery from Uncertain Data, pages 38–47, Paris, France, 2009.<br>[4] Xiufeng Jiang, Zhang Yi, and Jian Cheng Lv. Fuzzy svm with a new fuzzy membership function. Neural Computing &amp; Applications, 15(3-4):268–276, 2006.<br>[5]  Xin Geng, Qin Wang, and Yu Xia. Facial age estimation by adaptive label distribution learning. In Proceedings of the 22nd International Conference on Pattern Recognition, pages 4465–4470, Stockholm, Sweden, 2014.</p>]]></content>
      
      
      <categories>
          
          <category> Machine Learning </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Label-Specific Features for Multi-Label Classification</title>
      <link href="/2022/12/13/label-specific/"/>
      <url>/2022/12/13/label-specific/</url>
      
        <content type="html"><![CDATA[<p>如果对所有示例采用相同的特征集合来归纳分类模型，则会忽略每个标记的独特特征。如基于形状的特征能较好的区分物体，如猫和狗等，而基于颜色的特征则能更好的区分背景，如天空和大海等。</p><p>基于类属特征学习（label-sepcific feature learning）的目标在于，找到类别标记的相关并可区别特征[1]。</p><p>LIFT[2]: 对每个标记聚类，分析其中正类和负类示例。最后度量原始示例到聚类中心的距离，获取标记限定特征。<br>优化：降维，聚类，局部邻居信息，全局空间拓扑信息，相关标记信息</p><p>许多算法通过探索标记间的语义关系，来改进标记限定特征的学习。标记相关性常以先验知识的形式来对学习过程做限制[3-7]。这些算法计算成对标记之间的相似性，并将基于标记相关性的相似特性输入模型训练中，施加约束使强相关标记之间共享更多特征[8-10]或相似预测[11]。但是这些算法常采用预计算的相似矩阵，并不利于下游任务的学习。</p><h2 id="Collaborative-Learning-of-Label-Semantics-and-Deep-Label-Specific-Features-for-Multi-Label-Classification"><a href="#Collaborative-Learning-of-Label-Semantics-and-Deep-Label-Specific-Features-for-Multi-Label-Classification" class="headerlink" title="Collaborative Learning of Label Semantics and Deep Label-Specific Features for Multi-Label Classification"></a>Collaborative Learning of Label Semantics and Deep Label-Specific Features for Multi-Label Classification</h2><h3 id="Collaborative-learning"><a href="#Collaborative-learning" class="headerlink" title="Collaborative learning:"></a>Collaborative learning:</h3><ul><li>Label embedding: 选择标记最相关特征</li><li>Discrimination errors: 通过反向传播，识别标记限定特征的判别属性</li></ul><h3 id="Label-semantic-encoding"><a href="#Label-semantic-encoding" class="headerlink" title="Label semantic encoding"></a>Label semantic encoding</h3><p>采用GIN获得标记共现的语义空间的标记嵌入。</p><ul><li><p>共现矩阵：</p><script type="math/tex; mode=display">A_{ij} = \frac{1}{2} [P(l_j|l_i)+P(l_i|l_j)]</script></li><li><p>GIN优化标记嵌入矩阵：</p><script type="math/tex; mode=display">H^{t+1} = f^{t+1}[(1+\epsilon^{t+1})H^{t}+AH^{t}]</script></li><li><p>pairwise decoder 区别非邻相关标记：</p><script type="math/tex; mode=display">L_{le} = \frac{1}{q^2} \sum_{i=1}^q \sum_{j=1}^q [cos(e_i,e_j)-\hat{A_{ij}}]^2</script></li></ul><h3 id="Semantic-Guided-Feature-Disentangling"><a href="#Semantic-Guided-Feature-Disentangling" class="headerlink" title="Semantic-Guided Feature-Disentangling"></a>Semantic-Guided Feature-Disentangling</h3><p>标记语义为类属特征的学习提供指导以及标记关联信息。</p><ul><li>类属特征映射：<script type="math/tex; mode=display">\phi_k(x) = \phi(x,e_k)</script>这个过程挺简单的，就是一个全连接层+attention，采用了leakyReLU。</li></ul><p>参考文献:<br>[1] Hang J Y, Zhang M L. Collaborative Learning of Label Semantics and Deep Label-Specific Features for Multi-Label Classification[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.<br>[2] Zhang M L, Wu L. Lift: Multi-label learning with label-specific features[J]. IEEE transactions on pattern analysis and machine intelligence, 2014, 37(1): 107-120.<br>[3] J.-H. Ma and T. Chow, “Topic-based instance and feature selection in multilabel classification,” IEEE Trans. Neural Netw. Learn. Syst., early access, Oct. 27, 2020, doi: 10.1109/TNNLS.2020.3027745.<br>[4] X.-Y. Che, D.-G. Chen, and J.-S. Mi, “A novel approach for learning label correlation with application to feature selection of multilabel data,” Inf. Sci., vol. 512, pp. 795–812, 2020.<br>[5] H.-R. Han, M.-X. Huang, Y. Zhang, X.-G. Yang, and W.-L. Feng, “Multi-label learning with label specific features using correlation information,” IEEE Access, vol. 7, pp. 11 474–11 484, 2019.<br>[6] Z.-S. Chen and M.-L. Zhang, “Multi-label learning with regularization enriched label-specific features,” in Proc. 11th Asian Conf. Mach. Learn., Nagoya, Japan, 2019, pp. 411–424.<br>[7] J. Huang, G.-R. Li, Q.-M. Huang, and X.-D. Wu, “Learning label-specific features and class-dependent labels for multi-label classification,” IEEE Trans. Knowl. Data Eng., vol. 28, no. 12, pp. 3309–3323, Dec. 2016.<br>[8] J. Huang, G.-R. Li, Q.-M. Huang, and X.-D. Wu, “Learning label specific features for multi-label classification,” in Proc. 15th IEEE Int. Conf. Data Mining, Atlantic City, NJ, USA, 2015, pp. 181–190.<br>[9] J. Huang, G.-R. Li, Q.-M. Huang, and X.-D. Wu, “Joint feature selection and classification for multilabel learning,” IEEE Trans. Cybern., vol. 48, no. 3, pp. 876–889, 2018.<br>[10] J. Xu, H. Tian, Z. Wang, Y. Wang, W. Kang, and F. Chen, “Joint input and output space learning for multi-label image classification,” IEEE Trans. Multimedia, vol. 23, pp. 1696–1707, 2021.<br>[11] X.-Y. Jia, S.-S. Zhu, and W.-W. Li, “Joint label-specific features and correlation information for multi-label learning,” J. Comput. Sci. Technol., vol. 35, no. 2, pp. 247–258, 2020.</p>]]></content>
      
      
      <categories>
          
          <category> Machine Learning </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Multi-label learning</title>
      <link href="/2022/12/09/multi-label/"/>
      <url>/2022/12/09/multi-label/</url>
      
        <content type="html"><![CDATA[<p>多标记学习[4]的关键在于对标记间相关性的利用：</p><ul><li>一阶策略：忽略标记相关性。<ul><li>Binary Relevance[1]：将多标记问题拆分成多个二分类问题</li><li>ML-KNN</li></ul></li><li>二阶策略：考虑标记间的成对相似性。<ul><li>CML[2]：将多标记数据集转换为 <script type="math/tex">k(k-1)/2</script> 个成对比较的数据集，每个标记对看作是一个类别。把 <script type="math/tex">y_i</script> 是相关标记 <script type="math/tex">y_j</script> 是无关标记的样本看成正类，把 <script type="math/tex">y_i</script> 是无关标记 <script type="math/tex">y_j</script> 是相关标记的样本看成负类，这两个标记同为相关或无关标记的样本被抛弃。然后在每个数据集上训练一个二分类模型，预测时综合考虑多个与某标记有关的分类器，最终结果由多数投票决定。</li><li>Calibrated Label Ranking</li><li>Rank-SVM</li></ul></li><li>高阶策略：考虑标记间的高阶关系。<ul><li>Random k-labelsets[3]：先随机抽取一些只有k个标记的子集，然后用 Label Powerset 技术将每种组合看成一个新标记，然后就可以使用单标记学习算法进行学习。</li><li>LEAD</li></ul></li></ul><h2 id="学习方法"><a href="#学习方法" class="headerlink" title="学习方法"></a>学习方法</h2><ul><li>问题转换：利用现有的学习框架，将多标记学习问题转化为二分类或者排序问题，使数据适应算法<ul><li>Binaray Relevance, Random k-labelsets</li><li>Calibrated Label Ranking[5]: 有点类似于CML</li></ul></li><li>算法适应：对现有算法进行改造，使算法适应数据<ul><li>Multi-label k-nearest neighbor(ML-KNN), Multi-label decision tree(ML-DT)</li></ul></li></ul><p>参考文献：<br>[1] R.Boutell, Matthew, Luo, Jiebo, Shen, Xipeng, et al. Learning multi-label scene classification[J]. Pattern recognition, 2004, 37(9):1757–1771.<br>[2] Ghamrawi, Nadia and McCallum, Andrew. Collective multi-label classification[C]. In: Proceedings of the ACM International Conference on Information and Knowledge Management. 2005. 195–200.<br>[3] Tsoumakas, Grigorios and Vlahavas, Ioannis. Random k-labelsets: An ensemble method for multilabel classification[C]. In: Proceedings of the European Conference on Machine Learning. 2007. 406–417.<br>[4] Zhang, Minling and Zhou, Zhihua. A review on multi-label learning algorithms[J]. IEEE Transactions on Knowledge and Data Engineering, 2014, 26(8):1819–1837.<br>[5] Johannes Fürnkranz, Eyke Hüllermeier, Eneldo Loza Mencía, and Klaus Brinker. Multilabel classification via calibrated label ranking. Machine learning, 73(2):133–153, 2008</p>]]></content>
      
      
      <categories>
          
          <category> Machine Learning </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Weakly supervised learning</title>
      <link href="/2022/12/08/partial/"/>
      <url>/2022/12/08/partial/</url>
      
        <content type="html"><![CDATA[<p>数据标注成本过高，导致数据不具备完全正确的监督信息。</p><p>弱监督学习可以划分为三类[1]：</p><ul><li>不完整（incomplete）的监督信息：数据集中只有部分（小部分）含有标记，大部分没有标记。</li><li>不精确（inexact）的监督信息：只有粗粒度的标记。</li><li>不正确（inaccurate）的监督信息：给的标记并不总是真实标记。</li></ul><h2 id="不完整（incomplete）的监督信息学习"><a href="#不完整（incomplete）的监督信息学习" class="headerlink" title="不完整（incomplete）的监督信息学习"></a>不完整（incomplete）的监督信息学习</h2><ul><li>半监督学习（semi-supervised learning）[2,3]：<ul><li>相似示例具有相似的类标记：聚类假设（cluster assumption）和流形假设（manifold assumption）。</li><li>半监督SVM，半监督图学习，协同训练（co-training）</li></ul></li><li>主动学习（active learning）:允许对少量无标记数据查询其正确的标记从而改善学习性能。<ul><li>挑选最少最有用的查询样本：挑选不确定性最高样本或差异性最大样本。<ul><li>不确定性样本：信息量大，查询这些样本可以降低模型的不确定程度。</li><li>差异性大样本：不重复，能够更完整的描述整个数据集。</li></ul></li></ul></li><li>PU学习（positive-unlabeled learning）[4,5]： 标记只有正样本，负样本过于昂贵，如临床数据。</li></ul><h2 id="不精确（inexact）的监督信息学习"><a href="#不精确（inexact）的监督信息学习" class="headerlink" title="不精确（inexact）的监督信息学习"></a>不精确（inexact）的监督信息学习</h2><ul><li>多示例学习（multi-instance learning）：多个示例共同组成一个包，如果这个包中有正样本，那么<br>这个包为正包，但并不知道哪个或哪些样本为正。</li><li>偏标记学习（partial-label learning）：也被称为模糊标记学习（ambiguously label learning）或超集合标记学习（superset label learning）。在偏标记学习中，每个样本对应于多个候选标记 (candidate label)，其中有且只有一个能描述该样本的真实语义信息，而这个正确的标记在学习过程中是未知的。<ul><li>基于辨识的策略（identification-based strategy）：也称为消歧（disambiguating），旨在在训练阶段启发式地对每个候选标记集进行净化，提取正确的标记来消除候选标记造成的监督不精确。<ul><li>其中所有候选标记是一种竞争关系，即每个标记对学习目标的影响是不同的，并且一个标记的影响变大，其他标记的影响则势必会变小。</li><li>基于辨识策略的算法在给每个候选标记打分 (score) 和通过优化某些特殊设计的评价准则之间迭代。候选标记的分数表示它是正确标记的可能性，训练完成时分数最大的认为是正确标记，通常用估计出来的类后验概率当作分数。</li><li>将正确标记看作隐变量，学习目标是直接最大化一个候选标记上的输出，将其作为正确标记。</li><li>最大似然，最大间隔，深度学习</li></ul></li><li>基于平均的策略（average-based strategy）:认为所有候选标记都是等价的，所有候选标记以协作的方式对学习目标造成影响，在训练过程中不需要识别出隐含的正确标记。在训练阶段将所有候选标记“一视同仁”，每个候选标记对学习目标造成的影响是相同的。</li><li>补标记学习（complementary-label learning）：补标记为每个示例指定一个它一定不属于的类，因此它可以被认为是一个极端情况下的<br>偏标记，每个示例与（类别数-1）个候选标记相对应。</li></ul></li><li>缺失标记学习（missing-label learning）[9,10]：示例与标记间的对应关系有所缺失，注意这种缺失没有数量限制，对缺失的是相关标记还是无关标记也没有限制，即，缺失“示例-标记”关系对。<ul><li>低秩假设（low rank assumption）[11,12]：所有示例的标记构成的矩阵是低秩的，即，假如标记空间有 <script type="math/tex">k</script> 个标记，那么可能出现的标记组合远远小于 <script type="math/tex">2^k</script> 个，这意味着有一些标记是强相关的，而有一些标记是矛盾的。<ul><li>映射模型[14,15]：恢复的标记矩阵是低秩的，同时满足和特征矩阵之间的映射关系。</li><li>关系模型[16]：假设缺失的标记和可见的标记间存在某种关系模型，将原始标记空间投影到低秩空间后再推断模型的参数。</li></ul></li><li>流形假设：相似特征的示例具有相近标记，利用特征空间的信息来对标记空间的缺失做一个补充。</li><li>低秩+流形[17,18]</li><li>模型优化[19,20]</li></ul></li></ul><h2 id="不正确（inaccurate）的监督信息学习"><a href="#不正确（inaccurate）的监督信息学习" class="headerlink" title="不正确（inaccurate）的监督信息学习"></a>不正确（inaccurate）的监督信息学习</h2><ul><li>噪声标记学习（noisy-label learning）：<ul><li>标记矫正[6,7]：通过一个干净的推断步骤来矫正错误标记</li><li>损失矫正[8]：修改损失函数来提高对错误标记的鲁棒性</li><li>改进训练策略：MentorNet &amp; StudentNet, co-teaching</li><li>对噪声鲁棒的损失函数</li></ul></li></ul><p>参考文献：<br>[1] Zhou, Zhihua. A brief introduction to weakly supervised learning[J]. National Science Review, 2017, 5(1):44–53.<br>[2] Zhu, Xiaojin J. Semi-supervised learning literature survey[J]. University of WisconsinMadison Department of Computer Sciences, 2005<br>[3] Chapelle, Olivier, Scholkopf, Bernhard, and Zien, Alexander. Semi-supervised learning[J]. IEEE Transactions on Neural Networks, 2009, 20(3):542–542.<br>[4] du Plessis, Marthinus C, Niu, Gang, and Sugiyama, Masashi. Analysis of learning from positive and unlabeled data[C]. In: Advances in neural information processing systems. 2014. 703–711.<br>[5] Kiryo, Ryuichi, Niu, Gang, du Plessis, Marthinus C, et al. Positive-unlabeled learning with non-negative risk estimator[C]. In: Advances in Neural Information Processing Systems. 2017. 1675–1685.<br>[6] Xiao, Tong, Xia, Tian, Yang, Yi, et al. Learning from massive noisy labeled data for image classification[C]. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015. 2691–2699.<br>[7] Veit, Andreas, Alldrin, Neil, Chechik, Gal, et al. Learning from noisy large-scale datasets with minimal supervision[C]. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. 839–847.<br>[8] Patrini, Giorgio, Rozza, Alessandro, Menon, Aditya Krishna, et al. Making deep neural networks robust to label noise: A loss correction approach[C]. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. 1944–1952<br>[9] Yu, Hsiang-Fu, Jain, Prateek, Kar, Purushottam, et al. Large-scale multi-label learning with missing labels[C]. In: Proceedings of the International Conference on Machine Learning. 2014. 593–601.<br>[10] Xu, Miao, Jin, Rong, and Zhou, Zhihua. CUR algorithm for partially observed matrices[C]. In: International Conference on Machine Learning. 2015. 1412–1421.<br>[11] Goldberg, Andrew, Recht, Ben, Xu, Junming, et al. Transduction with matrix completion: Three birds with one stone[J]. Advances in neural information processing systems, 2010, pages 757–765.<br>[12] Candes, Emmanuel J. and Recht, Benjamin. Exact matrix completion via convex optimization[J]. Foundations of Computational mathematics, 2009, 9(6):717–772.<br>[13] Jing, Liping, Yang, Liu, Yu, Jian, et al. Semi-supervised low-rank mapping learning for multi-label classification[C]. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015. 1483–1491.<br>[14] Yu, Hsiang-Fu, Jain, Prateek, Kar, Purushottam, et al. Large-scale multi-label learning<br>with missing labels[C]. In: Proceedings of the International Conference on Machine<br>Learning. 2014. 593–601.<br>[15] Xu, Linli, Wang, Zhen, Shen, Zefan, et al. Learning low-rank label correlations for multilabel classification with missing labels[C]. In: Proceedings of the IEEE International<br>Conference on Data Mining. 2014. 1067–1072.<br>[16] Bi, Wei and Kwok, James. Multilabel classification with label correlations and missing<br>labels[C]. In: Proceedings of the AAAI Conference on Artificial Intelligence. 2014.<br>1680–1686.<br>[17] Jing, Liping, Yang, Liu, Yu, Jian, et al. Semi-supervised low-rank mapping learning for multi-label classification[C]. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recog.<br>[18] Zhao, Feipeng and Guo, Yuhong. Semi-supervised multi-label learning with incomplete labels[C]. In: Proceedings of the International Joint Conference on Artificial Intelligence. 2015. 4062–4068.<br>[19] Xu, Miao, Jin, Rong, and Zhou, Zhihua. CUR algorithm for partially observed matrices[C]. In: International Conference on Machine Learning. 2015. 1412–1421.<br>[20] Xu, Miao, Jin, Rong, and Zhou, Zhihua. Speedup matrix completion with side information: Application to multi-label learning[C]. In: Advances in neural information processing systems. 2013. 2301–2309.</p>]]></content>
      
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>BERT</title>
      <link href="/2022/12/02/BERT/"/>
      <url>/2022/12/02/BERT/</url>
      
        <content type="html"><![CDATA[<p>参考资料：<a href="https://github.com/google-research/bert">BERT(Bidirectional Encoder Representations from Transformers)</a>，<a href="https://huggingface.co/docs/transformers/index" target="_blank" rel="noopener">HUGGING FACE</a>，<a href="https://www.ylkz.life/deeplearning/p10237847/" target="_blank" rel="noopener">月来客栈</a></p><ul><li><p>数据预处理</p><ul><li><p>文本格式化</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">def read_wiki2(filepath=None):</span><br><span class="line">    with open(filepath, &apos;r&apos;) as f:</span><br><span class="line">        lines = f.readlines()  # 一次读取所有行，每一行为一个段落</span><br><span class="line">    paragraphs = []</span><br><span class="line">    for line in tqdm(lines, ncols=80, desc=&quot; ## 正在读取原始数据&quot;):</span><br><span class="line">        if len(line.split(&apos; . &apos;)) &gt;= 2:</span><br><span class="line">            paragraphs.append(line.strip().lower().split(&apos; . &apos;))</span><br><span class="line">    random.shuffle(paragraphs)  # 将所有段落打乱</span><br><span class="line">    return paragraphs</span><br></pre></td></tr></table></figure></li><li></li></ul></li></ul>]]></content>
      
      
      <categories>
          
          <category> Machine Learning </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
            <tag> BERT </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Med BERT</title>
      <link href="/2022/10/26/med-bert/"/>
      <url>/2022/10/26/med-bert/</url>
      
        <content type="html"><![CDATA[<p>最近准备使用<a href="https://github.com/ZhiGroup/Med-BERT">Med-BERT</a>对结构化EHR数据做Embedding。<br>在论文《Med-BERT》中的引用为<a href="https://github.com/ZhiGroup/pytorch_ehr">pytorch_ehr</a>。</p><p>Med-BERT主要还是修改了<a href="https://github.com/google-research/bert">BERT</a>。包含两个文件夹<code>Fine-Tunning Tutorials</code>和<code>Pretraining Code</code>。</p><ul><li>Pretraining Code:<ul><li><code>preprocess_pretrain_data.py</code>: 处理<code>.csv</code>中的患者数据。主要是提取了患者就诊的时间及间隔，疾病代码，并编码。<ul><li>input: <code>data_file</code>, <code>vocab</code>, <code>output_file</code>,<code>data_size</code></li><li>output: <ul><li><code>.types</code> : vocab</li><li><code>.ptencs</code>: (pts_ls) patients encounter # [住院时间，下一次住院间隔时间，ICD代码，编码后的ICD代码]</li><li><code>.encs</code>: tokenized encounter data and labels # 代码中没有看到</li><li><code>.bencs</code>: （pts_sls）tokeninzed encounter data and labels, with other segments # [患者ID，住院时间，下一次住院时间，编码后的ICD代码，就医次数]</li></ul></li></ul></li><li><code>create_BERTpretrain_EHRfeatures.py</code>：对应于BERT中的<code>create_pretraining_data.py</code>，主要是将<code>.bencs</code>中的代码序列写为TF格式，并构建MASK。<ul><li>input: <code>.bencs.train</code>, <code>output_file</code>, <code>.types</code></li><li>function:<ul><li><code>write_EHRinstance_to_example_file</code>:<ul><li>input: <code>train_data(.bencs), max_seq_length, max_predictions_per_seq, masked_lm_prob, vocab, output_files, rng</code></li><li>output:<code>features(input_ids, input_mask, segment_ids, masked_lm_positions, masked_lm_ids, maked_lm_weights, next_sentence_labels)</code></li></ul></li><li><code>create_masked_EHR_predictions</code><ul><li>input: <code>input_seq(diag_lm), masked_lm_prob, max_predictions_pre_seq, vocab, rng</code></li><li>output: <code>input_ids(masked_token), masked_lm_positions, masked_lm_ids</code></li></ul></li></ul></li></ul></li><li><code>run_EHRpretraining.py</code>: 对应于<code>run_pretraining.py</code>，采用BERT预训练。<ul><li>input: <code>bert_config_file</code>, <code>input_file</code>, <code>output_dir</code>, etc.</li><li>functions:<ul><li><code>model_fn_builder</code>:<ul><li>input: <code>bert_config, init_checkpoint, learning_rate, num_train_steps, etc)</code></li><li><code>model_fn</code>: <ul><li><code>model = modeling.BertModel()</code></li><li><code>get_masked_lm_output</code></li><li><code>get_next_sentence_output</code></li><li><code>optimization.create_optimizer</code></li><li><code>metric_fn</code> </li></ul></li></ul></li><li><code>input_fn_builder</code>: <ul><li>do_train: <code>is_training=True</code></li><li>do_eval: <code>is_training=False</code></li></ul></li></ul></li></ul></li></ul></li><li><code>Fine-Tunning Tutorials</code>: 主要由pytorch实现。<ul><li><code>Predicting_DHF_MED_BERT_LR.ipynb</code></li><li><code>Predicting_PC_RNN_BERT_pytorch_using_FT_Med-BERT.ipynb</code></li><li><code>Create_pretrainingEHR_dataFTv1.py</code>    </li></ul></li></ul><p>但是，pretrain的代码会报错，fine-tunning的没有给预训练模型。</p><p>所以还是尝试一下自己用bert模型做embedding。<br>参考资料：</p><ol><li><a href="https://blog.csdn.net/qq_36618444/article/details/124990851" target="_blank" rel="noopener">在自己的数据集上重新训练BERT</a></li><li><a href="https://zhuanlan.zhihu.com/p/391814780" target="_blank" rel="noopener">在BERT模型中添加自己的词汇</a></li><li><a href="https://blog.csdn.net/qq_36618444/article/details/124990851" target="_blank" rel="noopener">在自己的数据集上重新训练BERT</a></li><li><a href="https://huggingface.co/docs/transformers/index" target="_blank" rel="noopener">Huggingface</a></li></ol><p>详细内容记录在下一篇<a href="https://sceliay.github.io/2022/12/02/BERT/" target="_blank" rel="noopener">BERT</a>中。</p>]]></content>
      
      
      <categories>
          
          <category> AI for Medicine </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
            <tag> BERT </tag>
            
            <tag> Medicine </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Tensorflow模型</title>
      <link href="/2022/10/11/tensorflow-1/"/>
      <url>/2022/10/11/tensorflow-1/</url>
      
        <content type="html"><![CDATA[<p>整理一下, Tensorflow的模型构建，主要有sequential方法，subclass模型和函数式API。以及模型的嵌套、多输入输出以及自定义方法。</p><ol><li><p><a href="https://www.tensorflow.org/tutorials/quickstart/beginner" target="_blank" rel="noopener">简单模型</a></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><span class="line">import tensorflow as tf</span><br><span class="line"></span><br><span class="line"># 加载数据集</span><br><span class="line">mnist = tf.keras.datasets.mnist</span><br><span class="line"></span><br><span class="line">(x_train, y_train), (x_test, y_test) = mnist.load_data()</span><br><span class="line">x_train, x_test = x_train / 255.0, x_test / 255.0</span><br><span class="line"></span><br><span class="line"># sequential 模型</span><br><span class="line">model = tf.keras.models.Sequential([</span><br><span class="line">  tf.keras.layers.Flatten(input_shape=(28, 28)),</span><br><span class="line">  tf.keras.layers.Dense(128, activation=&apos;relu&apos;),</span><br><span class="line">  tf.keras.layers.Dropout(0.2),</span><br><span class="line">  tf.keras.layers.Dense(10)</span><br><span class="line">])</span><br><span class="line"></span><br><span class="line"># 损失函数</span><br><span class="line">loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)</span><br><span class="line"></span><br><span class="line"># 编译模型</span><br><span class="line">model.compile(optimizer=&apos;adam&apos;,</span><br><span class="line">              loss=loss_fn,</span><br><span class="line">              metrics=[&apos;accuracy&apos;])</span><br><span class="line"></span><br><span class="line"># 模型训练</span><br><span class="line">model.fit(x_train, y_train, epochs=5)</span><br><span class="line"></span><br><span class="line"># 性能检验</span><br><span class="line">model.evaluate(x_test,  y_test, verbose=2)</span><br></pre></td></tr></table></figure></li><li><p>其他</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"># 模型输出</span><br><span class="line">predictions = model(x_train[:1]).numpy()</span><br><span class="line"></span><br><span class="line"># softmax</span><br><span class="line">tf.nn.softmax(predictions).numpy()</span><br><span class="line"></span><br><span class="line"># 封装</span><br><span class="line">probability_model = tf.keras.Sequential([</span><br><span class="line">  model,</span><br><span class="line">  tf.keras.layers.Softmax()</span><br><span class="line">])</span><br></pre></td></tr></table></figure></li><li><p>复杂模型</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br></pre></td><td class="code"><pre><span class="line">import tensorflow as tf</span><br><span class="line">print(&quot;TensorFlow version:&quot;, tf.__version__)</span><br><span class="line"></span><br><span class="line">from tensorflow.keras.layers import Dense, Flatten, Conv2D</span><br><span class="line">from tensorflow.keras import Model</span><br><span class="line"></span><br><span class="line"># 加载数据</span><br><span class="line">mnist = tf.keras.datasets.mnist</span><br><span class="line"></span><br><span class="line">(x_train, y_train), (x_test, y_test) = mnist.load_data()</span><br><span class="line">x_train, x_test = x_train / 255.0, x_test / 255.0</span><br><span class="line"></span><br><span class="line"># Add a channels dimension</span><br><span class="line">x_train = x_train[..., tf.newaxis].astype(&quot;float32&quot;)</span><br><span class="line">x_test = x_test[..., tf.newaxis].astype(&quot;float32&quot;)</span><br><span class="line"></span><br><span class="line"># 切分数据batch以及shuffle</span><br><span class="line">train_ds = tf.data.Dataset.from_tensor_slices(</span><br><span class="line">    (x_train, y_train)).shuffle(10000).batch(32)</span><br><span class="line"></span><br><span class="line">test_ds = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(32)</span><br><span class="line"></span><br><span class="line"># subclass模型</span><br><span class="line">class MyModel(Model):</span><br><span class="line">  def __init__(self):</span><br><span class="line">    super(MyModel, self).__init__()</span><br><span class="line">    self.conv1 = Conv2D(32, 3, activation=&apos;relu&apos;)</span><br><span class="line">    self.flatten = Flatten()</span><br><span class="line">    self.d1 = Dense(128, activation=&apos;relu&apos;)</span><br><span class="line">    self.d2 = Dense(10)</span><br><span class="line"></span><br><span class="line">  def call(self, x):</span><br><span class="line">    x = self.conv1(x)</span><br><span class="line">    x = self.flatten(x)</span><br><span class="line">    x = self.d1(x)</span><br><span class="line">    return self.d2(x)</span><br><span class="line"></span><br><span class="line"># Create an instance of the model</span><br><span class="line">model = MyModel()</span><br><span class="line"></span><br><span class="line"># 损失函数以及优化器</span><br><span class="line">loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)</span><br><span class="line"></span><br><span class="line">optimizer = tf.keras.optimizers.Adam()</span><br><span class="line"></span><br><span class="line"># 度量损失与准确率</span><br><span class="line">train_loss = tf.keras.metrics.Mean(name=&apos;train_loss&apos;)</span><br><span class="line">train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name=&apos;train_accuracy&apos;)</span><br><span class="line"></span><br><span class="line">test_loss = tf.keras.metrics.Mean(name=&apos;test_loss&apos;)</span><br><span class="line">test_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name=&apos;test_accuracy&apos;)</span><br><span class="line"></span><br><span class="line"># 模型训练</span><br><span class="line">@tf.function</span><br><span class="line">def train_step(images, labels):</span><br><span class="line">  with tf.GradientTape() as tape:</span><br><span class="line">    # training=True is only needed if there are layers with different</span><br><span class="line">    # behavior during training versus inference (e.g. Dropout).</span><br><span class="line">    predictions = model(images, training=True)</span><br><span class="line">    loss = loss_object(labels, predictions)</span><br><span class="line">  gradients = tape.gradient(loss, model.trainable_variables)</span><br><span class="line">  optimizer.apply_gradients(zip(gradients, model.trainable_variables))</span><br><span class="line"></span><br><span class="line">  train_loss(loss)</span><br><span class="line">  train_accuracy(labels, predictions)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"># 测试模型</span><br><span class="line">@tf.function</span><br><span class="line">def test_step(images, labels):</span><br><span class="line">  # training=False is only needed if there are layers with different</span><br><span class="line">  # behavior during training versus inference (e.g. Dropout).</span><br><span class="line">  predictions = model(images, training=False)</span><br><span class="line">  t_loss = loss_object(labels, predictions)</span><br><span class="line"></span><br><span class="line">  test_loss(t_loss)</span><br><span class="line">  test_accuracy(labels, predictions)</span><br><span class="line"></span><br><span class="line"># 训练</span><br><span class="line">EPOCHS = 5</span><br><span class="line"></span><br><span class="line">for epoch in range(EPOCHS):</span><br><span class="line">  # Reset the metrics at the start of the next epoch</span><br><span class="line">  train_loss.reset_states()</span><br><span class="line">  train_accuracy.reset_states()</span><br><span class="line">  test_loss.reset_states()</span><br><span class="line">  test_accuracy.reset_states()</span><br><span class="line"></span><br><span class="line">  for images, labels in train_ds:</span><br><span class="line">    train_step(images, labels)</span><br><span class="line"></span><br><span class="line">  for test_images, test_labels in test_ds:</span><br><span class="line">    test_step(test_images, test_labels)</span><br><span class="line"></span><br><span class="line">  print(</span><br><span class="line">    f&apos;Epoch &#123;epoch + 1&#125;, &apos;</span><br><span class="line">    f&apos;Loss: &#123;train_loss.result()&#125;, &apos;</span><br><span class="line">    f&apos;Accuracy: &#123;train_accuracy.result() * 100&#125;, &apos;</span><br><span class="line">    f&apos;Test Loss: &#123;test_loss.result()&#125;, &apos;</span><br><span class="line">    f&apos;Test Accuracy: &#123;test_accuracy.result() * 100&#125;&apos;</span><br><span class="line">  )</span><br></pre></td></tr></table></figure></li><li><p><a href="https://www.tensorflow.org/guide/keras/functional" target="_blank" rel="noopener">函数式API</a></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line">import numpy as np</span><br><span class="line">import tensorflow as tf</span><br><span class="line">from tensorflow import keras</span><br><span class="line">from tensorflow.keras import layers</span><br><span class="line"></span><br><span class="line">inputs = keras.Input(shape=(784,))</span><br><span class="line"></span><br><span class="line">dense = layers.Dense(64, activation=&quot;relu&quot;)</span><br><span class="line">x = dense(inputs)</span><br><span class="line">x = layers.Dense(64, activation=&quot;relu&quot;)(x)</span><br><span class="line">outputs = layers.Dense(10)(x)</span><br><span class="line"></span><br><span class="line">model = keras.Model(inputs=inputs, outputs=outputs, name=&quot;mnist_model&quot;)</span><br><span class="line"></span><br><span class="line">(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()</span><br><span class="line"></span><br><span class="line">x_train = x_train.reshape(60000, 784).astype(&quot;float32&quot;) / 255</span><br><span class="line">x_test = x_test.reshape(10000, 784).astype(&quot;float32&quot;) / 255</span><br><span class="line"></span><br><span class="line">model.compile(</span><br><span class="line">    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),</span><br><span class="line">    optimizer=keras.optimizers.RMSprop(),</span><br><span class="line">    metrics=[&quot;accuracy&quot;],</span><br><span class="line">)</span><br><span class="line"></span><br><span class="line">history = model.fit(x_train, y_train, batch_size=64, epochs=2, validation_split=0.2)</span><br><span class="line"></span><br><span class="line">test_scores = model.evaluate(x_test, y_test, verbose=2)</span><br><span class="line">print(&quot;Test loss:&quot;, test_scores[0])</span><br><span class="line">print(&quot;Test accuracy:&quot;, test_scores[1])</span><br></pre></td></tr></table></figure></li><li><p>查看模型</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">model.summary()</span><br><span class="line"></span><br><span class="line"># 绘制计算图</span><br><span class="line">keras.utils.plot_model(model, &quot;my_first_model.png&quot;)</span><br><span class="line">keras.utils.plot_model(model, &quot;my_first_model_with_shape_info.png&quot;, show_shapes=True) #显示输入输出的形状</span><br></pre></td></tr></table></figure></li><li><p>模型存储与读取</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">model.save(&quot;path_to_my_model&quot;)</span><br><span class="line">del model</span><br><span class="line"># Recreate the exact same model purely from the file:</span><br><span class="line">model = keras.models.load_model(&quot;path_to_my_model&quot;)</span><br></pre></td></tr></table></figure></li><li><p>模型嵌套</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line">def get_model():</span><br><span class="line">    inputs = keras.Input(shape=(128,))</span><br><span class="line">    outputs = layers.Dense(1)(inputs)</span><br><span class="line">    return keras.Model(inputs, outputs)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">model1 = get_model()</span><br><span class="line">model2 = get_model()</span><br><span class="line">model3 = get_model()</span><br><span class="line"></span><br><span class="line">inputs = keras.Input(shape=(128,))</span><br><span class="line">y1 = model1(inputs)</span><br><span class="line">y2 = model2(inputs)</span><br><span class="line">y3 = model3(inputs)</span><br><span class="line">outputs = layers.average([y1, y2, y3])</span><br><span class="line">ensemble_model = keras.Model(inputs=inputs, outputs=outputs)</span><br></pre></td></tr></table></figure></li><li><p>多个输入输出的模型</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br></pre></td><td class="code"><pre><span class="line">num_tags = 12  # Number of unique issue tags</span><br><span class="line">num_words = 10000  # Size of vocabulary obtained when preprocessing text data</span><br><span class="line">num_departments = 4  # Number of departments for predictions</span><br><span class="line"></span><br><span class="line">title_input = keras.Input(</span><br><span class="line">    shape=(None,), name=&quot;title&quot;</span><br><span class="line">)  # Variable-length sequence of ints</span><br><span class="line">body_input = keras.Input(shape=(None,), name=&quot;body&quot;)  # Variable-length sequence of ints</span><br><span class="line">tags_input = keras.Input(</span><br><span class="line">    shape=(num_tags,), name=&quot;tags&quot;</span><br><span class="line">)  # Binary vectors of size `num_tags`</span><br><span class="line"></span><br><span class="line"># Embed each word in the title into a 64-dimensional vector</span><br><span class="line">title_features = layers.Embedding(num_words, 64)(title_input)</span><br><span class="line"># Embed each word in the text into a 64-dimensional vector</span><br><span class="line">body_features = layers.Embedding(num_words, 64)(body_input)</span><br><span class="line"></span><br><span class="line"># Reduce sequence of embedded words in the title into a single 128-dimensional vector</span><br><span class="line">title_features = layers.LSTM(128)(title_features)</span><br><span class="line"># Reduce sequence of embedded words in the body into a single 32-dimensional vector</span><br><span class="line">body_features = layers.LSTM(32)(body_features)</span><br><span class="line"></span><br><span class="line"># Merge all available features into a single large vector via concatenation</span><br><span class="line">x = layers.concatenate([title_features, body_features, tags_input])</span><br><span class="line"></span><br><span class="line"># Stick a logistic regression for priority prediction on top of the features</span><br><span class="line">priority_pred = layers.Dense(1, name=&quot;priority&quot;)(x)</span><br><span class="line"># Stick a department classifier on top of the features</span><br><span class="line">department_pred = layers.Dense(num_departments, name=&quot;department&quot;)(x)</span><br><span class="line"></span><br><span class="line"># Instantiate an end-to-end model predicting both priority and department</span><br><span class="line">model = keras.Model(</span><br><span class="line">    inputs=[title_input, body_input, tags_input],</span><br><span class="line">    outputs=[priority_pred, department_pred],</span><br><span class="line">)</span><br><span class="line"></span><br><span class="line">model.compile(</span><br><span class="line">    optimizer=keras.optimizers.RMSprop(1e-3),</span><br><span class="line">    loss=[</span><br><span class="line">        keras.losses.BinaryCrossentropy(from_logits=True),</span><br><span class="line">        keras.losses.CategoricalCrossentropy(from_logits=True),</span><br><span class="line">    ],</span><br><span class="line">    loss_weights=[1.0, 0.2],</span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"># 按名字分配</span><br><span class="line">model.compile(</span><br><span class="line">    optimizer=keras.optimizers.RMSprop(1e-3),</span><br><span class="line">    loss=&#123;</span><br><span class="line">        &quot;priority&quot;: keras.losses.BinaryCrossentropy(from_logits=True),</span><br><span class="line">        &quot;department&quot;: keras.losses.CategoricalCrossentropy(from_logits=True),</span><br><span class="line">    &#125;,</span><br><span class="line">    loss_weights=&#123;&quot;priority&quot;: 1.0, &quot;department&quot;: 0.2&#125;,</span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"># Dummy input data</span><br><span class="line">title_data = np.random.randint(num_words, size=(1280, 10))</span><br><span class="line">body_data = np.random.randint(num_words, size=(1280, 100))</span><br><span class="line">tags_data = np.random.randint(2, size=(1280, num_tags)).astype(&quot;float32&quot;)</span><br><span class="line"></span><br><span class="line"># Dummy target data</span><br><span class="line">priority_targets = np.random.random(size=(1280, 1))</span><br><span class="line">dept_targets = np.random.randint(2, size=(1280, num_departments))</span><br><span class="line"></span><br><span class="line">model.fit(</span><br><span class="line">    &#123;&quot;title&quot;: title_data, &quot;body&quot;: body_data, &quot;tags&quot;: tags_data&#125;,</span><br><span class="line">    &#123;&quot;priority&quot;: priority_targets, &quot;department&quot;: dept_targets&#125;,</span><br><span class="line">    epochs=2,</span><br><span class="line">    batch_size=32,</span><br><span class="line">)</span><br></pre></td></tr></table></figure></li><li><p>自定义损失函数</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line">def custom_mean_squared_error(y_true, y_pred):</span><br><span class="line">    return tf.math.reduce_mean(tf.square(y_true - y_pred))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">model = get_uncompiled_model()</span><br><span class="line">model.compile(optimizer=keras.optimizers.Adam(), loss=custom_mean_squared_error)</span><br><span class="line"></span><br><span class="line"># We need to one-hot encode the labels to use MSE</span><br><span class="line">y_train_one_hot = tf.one_hot(y_train, depth=10)</span><br><span class="line">model.fit(x_train, y_train_one_hot, batch_size=64, epochs=1)</span><br><span class="line"></span><br><span class="line"># 使用其他参数</span><br><span class="line">class CustomMSE(keras.losses.Loss):</span><br><span class="line">    def __init__(self, regularization_factor=0.1, name=&quot;custom_mse&quot;):</span><br><span class="line">        super().__init__(name=name)</span><br><span class="line">        self.regularization_factor = regularization_factor</span><br><span class="line"></span><br><span class="line">    def call(self, y_true, y_pred):</span><br><span class="line">        mse = tf.math.reduce_mean(tf.square(y_true - y_pred))</span><br><span class="line">        reg = tf.math.reduce_mean(tf.square(0.5 - y_pred))</span><br><span class="line">        return mse + reg * self.regularization_factor</span><br><span class="line"></span><br><span class="line">model = get_uncompiled_model()</span><br><span class="line">model.compile(optimizer=keras.optimizers.Adam(), loss=CustomMSE())</span><br><span class="line"></span><br><span class="line">y_train_one_hot = tf.one_hot(y_train, depth=10)</span><br><span class="line">model.fit(x_train, y_train_one_hot, batch_size=64, epochs=1)</span><br></pre></td></tr></table></figure></li></ol>]]></content>
      
      
      <categories>
          
          <category> Tensorflow </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
            <tag> Tensorflow </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Lable Distribution for Multimodal Machine Learning</title>
      <link href="/2022/04/18/mldl/"/>
      <url>/2022/04/18/mldl/</url>
      
        <content type="html"><![CDATA[<p><a href="https://link.springer.com/article/10.1007/s11704-021-0611-6" target="_blank" rel="noopener">全文链接</a></p><h2 id="导读"><a href="#导读" class="headerlink" title="导读"></a>导读</h2><p>多模态机器学习致力于通过多源数据来了解世界，如图像、语音和文本等。相比于单模态，多模态数据包含更多的信息，且模态之间具有互补性，因此多模态机器学习模型往往表现出比单模态更好的性能。然而，如何有效进行多模态融合，挖掘其中深层的特征表达，始终是多模态机器学习的重难点。区别于以往的多模态融合方法，本文考虑边信息引导多模态融合。提出多模态标记分布学习框架，利用边信息恢复出多模态标记分布，表示每个模态在描述示例时所占比例，然后采用多模态标记分布指导多模态融合，从而更准确的获取融合特征。针对不同的时序数据，本文进一步提出两个多模态标记分布学习算法，并分别应用于多模态情感识别与疾病诊断任务。实验结果正式了该算法优于现有先进算法。</p><h2 id="多模态标记分布学习"><a href="#多模态标记分布学习" class="headerlink" title="多模态标记分布学习"></a>多模态标记分布学习</h2><p>区别于以往的多模态机器学习算法仅考虑各个模态特征，本文认为多模态融合会受到环境影响，如图1所示，当视频处于昏暗的环境中，我们会更关注于音频与字幕，而当视频处于嘈杂且没有字幕的时候，我们会更关注于图像来分析视频的内容。因此，本文提出利用边信息恢复出多模态标记分布，用于表示各个模态在环境影响下对示例的描述程度，并用多模态标记分布指导多模态融合，从而获得更准确的融合特征。由于真实世界数据中，缺少多模态标记分布的确切标注，因此，我们提出了一个端到端的多模态标记分布学习框架，利用任务的监督信号，通过反馈学习来获得多模态标记分布。</p><p><img src="/img/example.png" alt="图1. 模态重要性示意图">。在不同环境下，各个模态在描述示例时的比重会发生变化。</p><p>定义 <script type="math/tex">\mathcal{X} = \mathbb{R}^{r_s} \times \mathbb{R}^{r_1} \times \mathbb{R}^{r_2} \times \cdots \times \mathbb{R}^{r_M}</script> 为输入空间，包括边信息（<script type="math/tex">r_s</script>维）与 <script type="math/tex">M</script> 个模态特征（用 <script type="math/tex">r_m</script> 表示第 <script type="math/tex">m</script> 个模态维度）。用 <script type="math/tex">\mathcal{Y} = \{y_j\}^q_{j=1}</script> 表示标记空间，共 <script type="math/tex">q</script> 个类。给定训练集 <script type="math/tex">\mathcal{D} = \{(\textbf{x}_i,\textbf{y}_i)|1 \le i \le N \}</script>，其中 <script type="math/tex">\textbf{x}_i = [ \textbf{s}_i, \textbf{x}_i^1, \textbf{x}_i^2, \cdots, \textbf{x}_i^M] \in \mathcal{X}</script> 为特征向量，包括边信息 <script type="math/tex">\textbf{s}_i</script> 与模态特征 <script type="math/tex">\textbf{x}_i^m</script>， <script type="math/tex">\textbf{y}_i \subseteq \mathcal{Y}</script> 为相应标记，<script type="math/tex">N</script>为训练集数量。则多模态标记分布学习的任务为学习一个预测模型：<script type="math/tex">f: \mathcal{X} \to 2^{\mathcal{Y}}</script> 。</p><p><img src="/img/mldl.png" alt="图2. 多模态标记分布学习框架。"></p><p>多模态标记分布学习框架如图2所示。定义 <script type="math/tex">d_{\textbf{s}}^{m} \in \mathbb{R}</script> 为第 <script type="math/tex">m</script> 个模态的描述度，则多模态标记分布为 <script type="math/tex">\textbf{d} = \{ d_{\textbf{s}}^1, d_{\textbf{s}}^{2}, \cdots, d_{\textbf{s}}^{M} \}</script>。我们假设 <script type="math/tex">d_{\textbf{s}}^{m} \in [0,1]</script>，那么所有模态可以完整描述该示例，即<script type="math/tex">\sum{ \textbf{d} } = 1</script>。由于描述度与概率分布具有相同的约束，因此 <script type="math/tex">d_{\textbf{s}}^{m}</script> 可以表示为条件概率，<script type="math/tex">d_{\textbf{s}}^{m} = P( \textbf{x}^m| \textbf{s})</script>。本文构建条件概率函数 <script type="math/tex">p(\textbf{x}^m|\textbf{s};\textbf{w})</script> 来求解：</p><script type="math/tex; mode=display">\textbf{d} = p(\textbf{x}|\textbf{s};\textbf{w}) \\ = softmax(\textbf{h}_{\textbf{d}} \textbf{w}_{\textbf{d}} + b_{\textbf{d}})</script><p>其中，<script type="math/tex">\textbf{w}_{\textbf{d}} \in \mathbb{R}^{r_{h,d} \times M}</script>为模型参数， <script type="math/tex">b_{\textbf{d}} \in \mathbb{R}</script>为偏置，<script type="math/tex">\textbf{h}_{\textbf{d}} \in \mathbb{R}^{r_{h,d}}</script>为隐含层输出，本文设置为两层隐含层：</p><script type="math/tex; mode=display">\textbf{h}_{\textbf{d}} = \sigma( \textbf{h}_{\textbf{d},1} \textbf{w}_{\textbf{h},\textbf{d}} + b_{\textbf{h},\textbf{d}}) \\\textbf{h}_{\textbf{d},1} = \sigma( \textbf{s}_a \textbf{w}_{\textbf{h},\textbf{d},1} + b_{\textbf{h},\textbf{d},1})</script><p>其中， <script type="math/tex">\textbf{w}_{\textbf{h},\textbf{d}} \in \mathbb{R}^{r_{h,d,1} \times r_{h,d}}$， $\textbf{w}_{\textbf{h},\textbf{d},1} \in \mathbb{R}^{r_s \times r_{h,d,1}}</script>，<script type="math/tex">\sigma</script>为激活函数，可以为任意非线性函数如tanh或relu等。为了获得更好的理解能力，本文采用了自注意力机制：</p><script type="math/tex; mode=display">\textbf{s}_a = softmax(\frac{QK^T}{\sqrt{r_s}})V\\Q = \textbf{s} \textbf{w}_Q\\K = \textbf{s} \textbf{w}_K\\V = \textbf{s} \textbf{w}_V</script><p>其中，<script type="math/tex">Q, K \in \mathbb{R}^{r_k}</script>，<script type="math/tex">V \in \mathbb{R}^{r_s}</script>，<script type="math/tex">\textbf{w}_Q , \textbf{w}_K, \in \mathbb{R}^{r_s \times r_k}$ and $\textbf{w}_V \in \mathbb{R}^{r_s \times r_s}</script>为参数矩阵。 </p><p>在对多模态进行融合前，我们采用线性变换将各模态特征映射到同一特征空间：</p><script type="math/tex; mode=display">\widetilde{\textbf{x}}^m = \sigma( \textbf{x}^m \textbf{w}_{x,m}+b_{x,m})</script><p>再用多模态标记分布指导多模态融合：</p><script type="math/tex; mode=display">\textbf{x}_F = Concat( \textbf{d} \otimes \widetilde{\textbf{x}} )\\= [d_{\textbf{s}}^1 \widetilde{\textbf{x}}^1, d_{\textbf{s}}^2 \widetilde{\textbf{x}}^2, \cdots ,d_{\textbf{s}}^M \widetilde{\textbf{x}}^M ]</script><p>最后，我们通过前馈神经网络构建多模态融合特征到任务标记的映射：</p><script type="math/tex; mode=display">\hat{\textbf{y}} = sigmoid( \textbf{h}_{\textbf{y}} \textbf{w}_{\textbf{y}} +b_{\textbf{y}}) \\\textbf{h}_{\textbf{y}} = \sigma( \textbf{h}_{\textbf{y},1} \textbf{w}_{\textbf{h},{\textbf{y}}} + b_{\textbf{h},{\textbf{y}}}) \\\textbf{h}_{\textbf{y},1} = \sigma(\textbf{x}_F  \textbf{w}_{\textbf{h},\textbf{y},1} + b_{\textbf{h},\textbf{y},1})</script><p>则多模态标记分布学习的任务为最小化分类交叉熵损失：</p><script type="math/tex; mode=display">L = -\sum y\cdot log\hat{y}</script><h2 id="时序多模态标记分布学习"><a href="#时序多模态标记分布学习" class="headerlink" title="时序多模态标记分布学习"></a>时序多模态标记分布学习</h2><p>对于时序数据，其特征空间为 <script type="math/tex">\widehat{\mathcal{X}} = \mathbb{R}^{r_s \times T_s} \times \mathbb{R}^{r_1 \times T_1} \times \mathbb{R}^{r_2 \times T_2} \times \cdots \times \mathbb{R}^{r_M \times T_M}</script>，其中<script type="math/tex">T_s</script>表示边信息的时间序列，<script type="math/tex">T_m</script>则表示第 <script type="math/tex">m</script> 个模态的时间序列。本文根据时序数据特点，提出了先融合多模态标记分布学习和后融合标记分布学习算法。</p><p>对于各个模态数据可对齐，且边信息会随时发生改变的时序数据，即<script type="math/tex">T_s = T_1 = T_2 = \cdots = T_m</script>，如图1所示的视频数据，本文提出先融合多模态标记分布。首先将边信息与各个模态在时间轴上进行对齐，并根据多模态标记分布学习框架进行融合，然后采用双向LSTM对融合后的时序特征进行学习：</p><script type="math/tex; mode=display">\textbf{H} = BiLSTM(\textbf{X}_F)</script><p>其中，<script type="math/tex">\textbf{X}_F = [ \textbf{x}_F^1, \textbf{x}_F^2, \cdots, \textbf{x}_F^T]</script>为融合后的时序特征，利用注意力机制进行池化：</p><script type="math/tex; mode=display">\textbf{H}_a = \textbf{a} \textbf{H}  \\\textbf{a} = softmax( \textbf{H}\textbf{w}_a )</script><p>则融合后特征到任务标记的映射变为：</p><script type="math/tex; mode=display">\hat{\textbf{y}} = sigmoid( \textbf{h}_H \textbf{w}_{\hat{y}} +b_{\hat{y}}) \\\textbf{h}_H = \sigma( \textbf{h}_{H,1} \textbf{w}_{H} + b_{H}) \\\textbf{h}_{H,1} = \sigma( \textbf{H}_a \textbf{w}_{H,1} + b_{H,1})</script><p>而对于各个模态无法对齐，且边信息不会发生改变的时序数据，如多模态疾病预测问题中，我们可以将患者信息作为边信息，而其检查结果作为多模态特征，则在该次就诊时患者信息不会发生改变，且检查结果数据之间不存在对齐，本文提出了后融合多模态标记分布学习。首先采用双向LSTM对时序数据进行特征表达：</p><script type="math/tex; mode=display">\textbf{H}_m = BiLSTM(\textbf{X}_m), 1 \le m \le M</script><p>其中， <script type="math/tex">\textbf{X}_m = [\textbf{x}_m^1, \textbf{x}_m^2, \cdots, \textbf{x}_m^{T_m}]</script>为第<script type="math/tex">m</script>个模态的时序数据，同样采用注意力机制进行池化：</p><script type="math/tex; mode=display">\textbf{h}_{m,a} = \textbf{a}_m \textbf{H}_m\\\textbf{a}_{m} = softmax( \textbf{H}_m \textbf{w}_{a,m} )</script><p>则<script type="math/tex">\textbf{h}_{m,a}</script>可以替代多模态标记分布学习框架中的 <script type="math/tex">\textbf{x}_m</script>进行多模态融合。</p><h2 id="实验"><a href="#实验" class="headerlink" title="实验"></a>实验</h2><p>本文将先融合多模态标记分布学习算法应用于情感分析任务，并在CMU-MOSI与CMU-MOSEI数据集上进行实验，与现有先进算法进行比较，展现出了更好的鲁棒性。如图3所示，为部分实验结果，展示了随着环境变化，各个模态重要性发生改变。</p><p><img src="/img/results.png" alt="图3. 部分实验结果展示。"></p><p>本文将后融合多模态标记分布学习算法应用于疾病诊断任务，在MIMIC-III数据集上进行实验。采用Word2vec方法将结构化的患者特征与检查结果进行词嵌入，从而挖掘特征间的潜在联系。该算法与现有先进算法进行比较，表现出更高的准确性与泛化能力。</p>]]></content>
      
      
      <categories>
          
          <category> Machine Learning </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>大黄饲养笔记</title>
      <link href="/2021/08/12/cat/"/>
      <url>/2021/08/12/cat/</url>
      
        <content type="html"><![CDATA[<p>5月的时候，有个小男孩捡到了黄桃，但是室友不同意养，于是我就接手了。<br>当时的小黄桃估计才一个月，就我的一只手掌大小。</p><p>虽然以前养过aki，但毕竟她已经是一只成熟的小猫咪了，能自己照顾自己。我就放个水和粮，每天铲屎就好了。而且aki让我有一种猫咪是个黏人物种的错觉，因为她总是自觉的趴我腿上。我坐着打游戏的时候，还会强行往我怀里钻。养了黄桃以后我才知道，原来那样的甜妹是可遇不可求的。</p><p>一个月的时候，那个小男孩送猫的时候顺便送了一罐麦德氏的羊奶粉，高钙低敏那款。我喂的第一天就吐了。吓的不行，后来泡稀一点就好了。开始估计不太适应。后来一罐吃完了以后，我换了麦德氏的猫专用羊奶，傻猫直接拉肚子了，吓得我赶紧买了高钙低敏，结果因为快递员失误，那个奶粉寄了快一个月。期间没办法，还是给傻猫吃猫专用的，后面倒也没拉肚子了。</p><p>对于奶粉没有研究太多，就查了一下麦德氏也是一个老牌子了，评价还行。关键是，傻猫吃了大概一个半月，就怎么也不喝奶粉了。不管是猫专用的还是高钙低敏的，泡了也不喝。那会儿为了凑单，还买了两罐麦德氏的慕斯罐头，喂她倒是吃了。</p><p>但是，我也是后来开始研究猫咪食物的时候，才发现这个罐头是真的差。光胶就有3种，还加了大米粉和大豆油这种很让人迷惑的东西。是零食罐，不能替代主食，却卖到10元/罐，大概也就是卖给我这种新手小白了，请避坑！</p><p>那会儿觉得罐头贵，一天一罐喂不起，于是买了皇家猫离乳期幼猫全价奶糕。是的，我又踩坑了。虽然查了皇家是一个老牌国产猫粮，但是这个粮真的差到不行。我甚至没法在官网的商品介绍里面看到它的配料表，可见它是有多心虚。问了客服，倒是给了。配料表：鸭肉粉，鸡肉粉，大米，鸡油，谷朊粉（小麦中提取出来的天然蛋白质），小麦粉，宠物饲料复合调味料和啤酒酵母粉（都是诱食剂）。加了大豆油，和不清不楚的矿物质，最后还来一个酿酒酵母细胞壁，是真的为了骗猫吃下了不少功夫。非无谷，有致敏风险，粗蛋白质32%里面不少来源于植物，加了很多油，所以猫粮非常油。真实营养不足，请避坑！</p><p>但是很神奇的是，我的猫就是不吃。我不知道是喂罐头喂多了，还是她聪明会挑食。然后我就在胡歌的广告下，买了卫仕的冻干猫粮。看配料表其实看不出太多问题，直少无谷，40%的粗蛋白，虽然其中不少来源于植物。粗脂肪有点低，只有14%，不过傻猫好歹吃了。</p><p>傻猫总是乱吃东西，比如头发，我养的树皮直接啃，我有点大脚趾外翻，买了校正用的橡胶套，她也直接啃了！我震惊之余，也网上查了一下，说是微量元素缺乏导致的异食癖，还买了一堆卫士的微量元素片、维生素片、营养膏，不过喂到现在也没见好转，还是喜欢咬东西和乱吃东西。早上刚把吃进去的毛线吐出来了，不知道是不是昨天喂的猫草的功劳。倒是证实了皇家猫粮营养不足的问题吧。</p><p>7月的时候，算着傻猫已经3个多月了，就带去医院打疫苗。我问了两个朋友，他们互不认识，竟然很巧合的在同一家医院打的疫苗，我就带着傻猫去了。当时对医院还是蛮信任的。所以就咨询了一下医生猫粮的事，她非常嫌弃卫仕，表示都是植物蛋白。然后推荐了皇家，我当时对猫粮已经有所了解，但是听她这么一说，对自己的了解产生了怀疑，她表示皇家是安全粮。我有一个朋友喂的金素力高，问她怎么样，她表示和皇家差不多，我虽然有点怀疑，但还是很认真的听着。医生强烈的推荐了纽顿，其次是冠能，甚至在医院的柜子上拿了一包冠能的粮，第一个明晃晃的写着大米还是啥的谷物我记不清了，我当时就惊呆了，医生强行解释了一下，后来第二次带猫去打疫苗的时候，上面的冠能就换成了纽顿。医生还推荐了go，我就有点神奇，go不是前段时间才出事吗，医生说，就那一个批次，其他的没有，但反正我是不敢买的了。最后推荐的是纽翠斯，不过说这个油，容易黑下巴。</p><p>回去以后，我还是蛮信任医生的，买了纽顿的猫粮，不过当时也没有调查清楚，纽顿T系列的确还行，性价比很高，但是，我非常愚蠢的买了S系列的幼猫，非无谷，贪便宜的后果吧。买来以后，颗粒很大，傻猫并不太喜欢。就纽顿和卫仕混着喂。</p><p>然后我看到了医院公众号在卖皇家、纽顿的猫粮，合着医生其实是在打广告？</p><p>买纽顿的时候，商家非常善良的送了一袋猫条。某天我闲着无聊，给她喂了一根，没想到她很喜欢吃，根本停不下来，于是我又喂了一根。就这样，我比较高频的喂了几次猫条以后，傻猫彻底不吃猫粮了。就喂罐头会吃一点。比较庆幸，那会儿618抢了10罐巅峰的主食猫罐头。这个罐头还是不错的。所以吧，小猫还是别乱喂零食了，很容易导致挑食。我家的还是一只大橘，就离谱。</p><p>出了这件事以后，我才开始很认真的看了猫粮的测评，小红书、知乎，各种查资料。然后对猫粮、罐头、零食才有了比较全面的了解。还加了两个猫车群，买了淘宝的宠物卡，其实对于养猫的还是省了不少钱。不过猫车的麻烦就是，当年看到折扣很高的时候，就会忍不住想买。我给傻猫买了不少玩具，逗猫棒、磨牙棒、毛线球、转盘、猫抓盆、激光笔等等。利用率最高的是毛线球，每天晚上会拿球跟猫玩，把球丢出去，她去追，乐此不疲。试图让她把球叼回来没有成功。但是早上我还睡着，她会把球叼我手上。我不理她就钻被子里咬我。但是陪猫猫玩还是很有必要的。不然晚上睡觉的时候在床头蹦迪。</p><p>后来看了小红书，买了一堆猫粮的小样让傻猫自己挑，还是蛮有意思的。我买了百利幼猫粮、渴望鸡、、绿福摩、爱肯拿、法米娜石榴鸡幼和纽翠斯红肉。这几种都是网上评价不错的，无谷，营养价值比较好的。因为法米娜和纽翠斯是另一家店买的，第二天才到。当时傻猫已经好几天没吃猫粮了，先拿前四种加给她选，她都吃完了，不过首先吃完的是绿福摩，而且绿福摩的颗粒很小，比皇家猫奶糕都小很多，还是蛮适合幼猫的。后来为了多骗她吃猫粮，又玩了一次，加上了卫仕和纽顿，她也吃完了。第二天法米娜和纽翠斯到了，再给她吃，她就变心了，特别喜欢法米娜，法米娜颗粒比绿福摩大一点，但是比其他的小，看网上评论都不差，就有一个测评说法米娜缺维生素B，长期喂需要单独补充。渴望和爱肯拿在外观上是真的很像，不过我的猫都不是特别爱吃。</p><p>在买小样的时候，蹲到了一个蛮便宜的百利幼，所以就下单了，结果比小样还早到。傻猫也挺喜欢吃的，不过有看到测评说含钙和镁过多，后面会考虑换绿福摩或者法米娜。现在就喂百利幼加卫仕的冻干。皇家早就喂流浪猫吃完了。现在给流浪猫喂纽顿，但是傻猫这个小心眼，每次看到我在小碗里放纽顿，就会跑过来吃。总之，不给傻猫吃零食以后，她吃猫粮还蛮乖。胖了不少。偶尔会喂个罐头，毕竟买了，别浪费。还有挺多猫条，以后再说吧，出门会给流浪猫带两根。</p><p>关于猫粮，出过事的猫粮挺多。go不说了，就最近，影响也广。百利和纽翠斯有开袋见虫。纽顿也出事过，有猫吃呕吐的。</p><p>也有看到说营养膏、维生素片和微量元素片是智商税，但是医院对于补充营养还是会建议营养膏。</p>]]></content>
      
      
      <categories>
          
          <category> Notes </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Daily </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Keras 学习笔记</title>
      <link href="/2021/03/13/keras/"/>
      <url>/2021/03/13/keras/</url>
      
        <content type="html"><![CDATA[<p>参考资料：<a href="https://keras.io/zh/" target="_blank" rel="noopener">Keras中文文档</a>,<a href="https://keras.io/" target="_blank" rel="noopener">Keras英文文档</a>, <a href="https://blog.csdn.net/u014061630/article/details/81086564" target="_blank" rel="noopener">Keras教程</a>, <a href="https://www.w3cschool.cn/tensorflow_python/tensorflow_python-5dym2s6j.html" target="_blank" rel="noopener">W3CSCHOOL</a>,<a href="https://www.tensorflow.org/api_docs/python/tf/keras" target="_blank" rel="noopener">Tensorflow中文社区</a>,</p><ol><li><p><code>Sequential</code>顺序模型</p><ul><li><p>定义：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">from keras.models import Sequential</span><br><span class="line">model = Sequential()</span><br></pre></td></tr></table></figure></li><li><p>使用<code>add</code>堆叠模型：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">from keras.layers import Dense</span><br><span class="line">model.add(Dense(units=64, activation=&apos;relu&apos;, input_dim=100))</span><br><span class="line">model.add(Dense(units=10, activation=&apos;softmax&apos;))</span><br></pre></td></tr></table></figure><ul><li><code>activation</code>: 激活函数</li><li><code>kernel_regularizer</code>和<code>bias_regularizer</code>: 层的权重、偏差的正则化方法。</li><li><code>kernel_initializer</code>和<code>bias_initializer</code>: 层创建时，权值和偏差的初始化方法，默认为<code>Glorot uniform</code><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"># A linear layer with L1 regularization of factor 0.01 applied to the kernel matrix:</span><br><span class="line">layers.Dense(64, kernel_regularizer=keras.regularizers.l1(0.01))</span><br><span class="line"></span><br><span class="line"># A linear layer with L2 regularization of factor 0.01 applied to the bias vector:</span><br><span class="line">layers.Dense(64, bias_regularizer=keras.regularizers.l2(0.01))</span><br><span class="line"></span><br><span class="line"># A linear layer with a kernel initialized to a random orthogonal matrix:</span><br><span class="line">layers.Dense(64, kernel_initializer=&apos;orthogonal&apos;)</span><br><span class="line"></span><br><span class="line"># A linear layer with a bias vector initialized to 2.0s:</span><br><span class="line">layers.Dense(64, bias_initializer=keras.initializers.constant(2.0))</span><br></pre></td></tr></table></figure></li></ul></li><li><p>使用<code>compile</code>配置学习过程</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">model.compile(loss=&apos;categorical_crossentropy&apos;,</span><br><span class="line">              optimizer=&apos;sgd&apos;,</span><br><span class="line">              metrics=[&apos;accuracy&apos;])</span><br></pre></td></tr></table></figure></li><li><p>配置优化器</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">model.compile(loss=keras.losses.categorical_crossentropy,</span><br><span class="line">              optimizer=keras.optimizers.SGD(lr=0.01, momentum=0.9, nesterov=True))</span><br></pre></td></tr></table></figure></li><li><p>训练数据</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"># x_train 和 y_train 是 Numpy 数组</span><br><span class="line">model.fit(x_train, y_train, epochs=5, batch_size=32)</span><br></pre></td></tr></table></figure></li><li><p>评估模型</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">loss_and_metrics = model.evaluate(x_test, y_test, batch_size=128)</span><br></pre></td></tr></table></figure></li><li><p>预测</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">classes = model.predict(x_test, batch_size=128)</span><br></pre></td></tr></table></figure></li><li><p><a href="https://keras.io/getting-started/sequential-model-guide/" target="_blank" rel="noopener">更多实例</a></p></li></ul></li><li><p><code>Functional API</code>函数式API</p><ul><li>可用于定义更复杂的模型。</li><li>层可调用，返回值为一个tensor</li><li>输入tensors和输出tensors被用来定义一个<code>tf.keras.model</code>实例</li><li><p>训练方法与sequential一样</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">inputs = keras.Input(shape=(32,))  # Returns a placeholder tensor</span><br><span class="line"></span><br><span class="line"># A layer instance is callable on a tensor, and returns a tensor.</span><br><span class="line">x = keras.layers.Dense(64, activation=&apos;relu&apos;)(inputs)</span><br><span class="line">x = keras.layers.Dense(64, activation=&apos;relu&apos;)(x)</span><br><span class="line">predictions = keras.layers.Dense(10, activation=&apos;softmax&apos;)(x)</span><br><span class="line"></span><br><span class="line"># Instantiate the model given inputs and outputs.</span><br><span class="line">model = keras.Model(inputs=inputs, outputs=predictions)</span><br><span class="line"></span><br><span class="line"># The compile step specifies the training configuration.</span><br><span class="line">model.compile(optimizer=tf.train.RMSPropOptimizer(0.001),</span><br><span class="line">              loss=&apos;categorical_crossentropy&apos;,</span><br><span class="line">              metrics=[&apos;accuracy&apos;])</span><br><span class="line"></span><br><span class="line"># Trains for 5 epochs</span><br><span class="line">model.fit(data, labels, batch_size=32, epochs=5)</span><br></pre></td></tr></table></figure></li><li><p><a href="https://keras.io/zh/getting-started/functional-api-guide/" target="_blank" rel="noopener">更多实例</a></p></li></ul></li><li><p>模型的方法和属性</p><ul><li><code>model.layers</code> 是包含模型网络层的展平列表。</li><li><code>model.inputs</code> 是模型输入张量的列表。</li><li><code>model.outputs</code> 是模型输出张量的列表。</li><li><code>model.summary()</code> 打印出模型概述信息。 它是 utils.print_summary 的简捷调用。</li><li><code>model.get_config()</code> 返回包含模型配置信息的字典。通过以下代码，就可以根据这些配置信息重新实例化模型：</li><li><code>model.get_weights()</code> 返回模型中所有权重张量的列表，类型为 Numpy 数组。</li><li><code>model.set_weights(weights)</code> 从 Numpy 数组中为模型设置权重。列表中的数组必须与<code>get_weights()</code>返回的权重具有相同的尺寸。</li><li><code>model.save_weights(filepath)</code> 将模型权重存储为 HDF5 文件。</li><li><code>model.load_weights(filepath, by_name=False)</code>: 从 HDF5 文件（由 save_weights 创建）中加载权重。默认情况下，模型的结构应该是不变的。 如果想将权重载入不同的模型（部分层相同）， 设置 by_name=True 来载入那些名字相同的层的权重。</li></ul></li><li><p><code>Model</code>类继承<br>通过编写<code>tf.keras.Model</code>的子类来构建一个自定义模型。在<code>init</code>方法里创建 layers。在<code>call</code>方法里定义前向传播过程。在<code>call</code>中，你可以指定自定义的损失函数，通过调用<code>self.add_loss(loss_tensor)</code>。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line">import keras</span><br><span class="line"></span><br><span class="line">class SimpleMLP(keras.Model):</span><br><span class="line"></span><br><span class="line">    def __init__(self, use_bn=False, use_dp=False, num_classes=10):</span><br><span class="line">        super(SimpleMLP, self).__init__(name=&apos;mlp&apos;)</span><br><span class="line">        self.use_bn = use_bn</span><br><span class="line">        self.use_dp = use_dp</span><br><span class="line">        self.num_classes = num_classes</span><br><span class="line"></span><br><span class="line">        self.dense1 = keras.layers.Dense(32, activation=&apos;relu&apos;)</span><br><span class="line">        self.dense2 = keras.layers.Dense(num_classes, activation=&apos;softmax&apos;)</span><br><span class="line">        if self.use_dp:</span><br><span class="line">            self.dp = keras.layers.Dropout(0.5)</span><br><span class="line">        if self.use_bn:</span><br><span class="line">            self.bn = keras.layers.BatchNormalization(axis=-1)</span><br><span class="line"></span><br><span class="line">    def call(self, inputs):</span><br><span class="line">        x = self.dense1(inputs)</span><br><span class="line">        if self.use_dp:</span><br><span class="line">            x = self.dp(x)</span><br><span class="line">        if self.use_bn:</span><br><span class="line">            x = self.bn(x)</span><br><span class="line">        return self.dense2(x)</span><br><span class="line"></span><br><span class="line">model = SimpleMLP()</span><br><span class="line">model.compile(...)</span><br><span class="line">model.fit(...)</span><br></pre></td></tr></table></figure><p>在类继承模型中，模型的拓扑结构是由 Python 代码定义的（而不是网络层的静态图）。这意味着该模型的拓扑结构不能被检查或序列化。因此，以下方法和属性不适用于类继承模型：</p><ul><li><code>model.inputs</code> 和 <code>model.outputs</code>。</li><li><code>model.to_yaml()</code> 和 <code>model.to_json()</code>。</li><li><code>model.get_config()</code> 和 <code>model.save()</code>。</li></ul></li><li><p><a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers" target="_blank" rel="noopener">tf.keras.layers</a></p><ul><li><a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/Input" target="_blank" rel="noopener">Input</a>: 定义模型的输入</li><li><a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding" target="_blank" rel="noopener">Embedding</a>: 定义嵌入层[<a href="http://frankchen.xyz/2017/12/18/How-to-Use-Word-Embedding-Layers-for-Deep-Learning-with-Keras/" target="_blank" rel="noopener">参考</a>]<ul><li>Keras提供了一个嵌入层，适用于文本数据的神经网络。</li><li>它要求输入数据是整数编码的，所以每个字都用一个唯一的整数表示。这个数据准备步骤可以使用Keras提供的Tokenizer API来执行。</li><li>嵌入层用随机权重进行初始化，并将学习训练数据集中所有单词的嵌入。</li><li><code>e = Embedding(input_dim=200, output_dim=32, input_length=50)</code>  </li></ul></li><li><a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/add" target="_blank" rel="noopener">add</a>: 将两个输出加和</li><li><a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/concatenate" target="_blank" rel="noopener">Concatenate</a>: 链接两个张量</li><li><a href="https://www.programcreek.com/python/example/89694/keras.layers.dot" target="_blank" rel="noopener">dot</a></li></ul></li><li><p>自定义layer<br>可以通过编写<code>tf.keras.layers.Layer</code>的子类来创建一个自定义<code>layer</code>，该子类编写过程中需要编写下面的方法：</p><ul><li><code>build</code>：创建层的参数。通过<code>add_weight</code>来添加权值</li><li><code>call</code>：定义前向传播过程。</li><li><code>compute_output_shape</code>：指定怎么根据输入去计算<code>layer</code>的输出<code>shape</code>。</li><li>layer可以通过<code>get_config</code>方法和<code>from_config</code>方法实现串行。<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br></pre></td><td class="code"><pre><span class="line">class MyLayer(keras.layers.Layer):</span><br><span class="line"></span><br><span class="line">  def __init__(self, output_dim, **kwargs):</span><br><span class="line">    self.output_dim = output_dim</span><br><span class="line">    super(MyLayer, self).__init__(**kwargs)</span><br><span class="line"></span><br><span class="line">  def build(self, input_shape):</span><br><span class="line">    shape = tf.TensorShape((input_shape[1], self.output_dim))</span><br><span class="line">    # Create a trainable weight variable for this layer.</span><br><span class="line">    self.kernel = self.add_weight(name=&apos;kernel&apos;,</span><br><span class="line">                                  shape=shape,</span><br><span class="line">                                  initializer=&apos;uniform&apos;,</span><br><span class="line">                                  trainable=True)</span><br><span class="line">    # Be sure to call this at the end</span><br><span class="line">    super(MyLayer, self).build(input_shape)</span><br><span class="line"></span><br><span class="line">  def call(self, inputs):</span><br><span class="line">    return tf.matmul(inputs, self.kernel)</span><br><span class="line"></span><br><span class="line">  def compute_output_shape(self, input_shape):</span><br><span class="line">    shape = tf.TensorShape(input_shape).as_list()</span><br><span class="line">    shape[-1] = self.output_dim</span><br><span class="line">    return tf.TensorShape(shape)</span><br><span class="line"></span><br><span class="line">  def get_config(self):</span><br><span class="line">    base_config = super(MyLayer, self).get_config()</span><br><span class="line">    base_config[&apos;output_dim&apos;] = self.output_dim</span><br><span class="line"></span><br><span class="line">  @classmethod</span><br><span class="line">  def from_config(cls, config):</span><br><span class="line">    return cls(**config)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"># Create a model using the custom layer</span><br><span class="line">model = keras.Sequential([MyLayer(10),</span><br><span class="line">                          keras.layers.Activation(&apos;softmax&apos;)])</span><br><span class="line"></span><br><span class="line"># The compile step specifies the training configuration</span><br><span class="line">model.compile(optimizer=tf.train.RMSPropOptimizer(0.001),</span><br><span class="line">              loss=&apos;categorical_crossentropy&apos;,</span><br><span class="line">              metrics=[&apos;accuracy&apos;])</span><br><span class="line"></span><br><span class="line"># Trains for 5 epochs.</span><br><span class="line">model.fit(data, targets, batch_size=32, epochs=5)</span><br></pre></td></tr></table></figure></li></ul></li><li><p>LSTM</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br></pre></td><td class="code"><pre><span class="line">from keras.layers import Input, Embedding, LSTM, Dense</span><br><span class="line">from keras.models import Model</span><br><span class="line"></span><br><span class="line"># 标题输入：接收一个含有 100 个整数的序列，每个整数在 1 到 10000 之间。</span><br><span class="line"># 注意我们可以通过传递一个 &quot;name&quot; 参数来命名任何层。</span><br><span class="line">main_input = Input(shape=(100,), dtype=&apos;int32&apos;, name=&apos;main_input&apos;)</span><br><span class="line"></span><br><span class="line"># Embedding 层将输入序列编码为一个稠密向量的序列，</span><br><span class="line"># 每个向量维度为 512。</span><br><span class="line">x = Embedding(output_dim=512, input_dim=10000, input_length=100)(main_input)</span><br><span class="line"></span><br><span class="line"># LSTM 层把向量序列转换成单个向量，</span><br><span class="line"># 它包含整个序列的上下文信息</span><br><span class="line">lstm_out = LSTM(32)(x)</span><br><span class="line"></span><br><span class="line"># 插入辅助损失</span><br><span class="line">auxiliary_output = Dense(1, activation=&apos;sigmoid&apos;, name=&apos;aux_output&apos;)(lstm_out)</span><br><span class="line"></span><br><span class="line">auxiliary_input = Input(shape=(5,), name=&apos;aux_input&apos;)</span><br><span class="line">x = keras.layers.concatenate([lstm_out, auxiliary_input])</span><br><span class="line"></span><br><span class="line"># 堆叠多个全连接网络层</span><br><span class="line">x = Dense(64, activation=&apos;relu&apos;)(x)</span><br><span class="line">x = Dense(64, activation=&apos;relu&apos;)(x)</span><br><span class="line">x = Dense(64, activation=&apos;relu&apos;)(x)</span><br><span class="line"></span><br><span class="line"># 最后添加主要的逻辑回归层</span><br><span class="line">main_output = Dense(1, activation=&apos;sigmoid&apos;, name=&apos;main_output&apos;)(x)</span><br><span class="line"></span><br><span class="line"># 定义</span><br><span class="line">model = Model(inputs=[main_input, auxiliary_input], outputs=[main_output, auxiliary_output])</span><br><span class="line"></span><br><span class="line"># 编译</span><br><span class="line">model.compile(optimizer=&apos;rmsprop&apos;, loss=&apos;binary_crossentropy&apos;,</span><br><span class="line">              loss_weights=[1., 0.2])</span><br><span class="line"></span><br><span class="line"># 传递输入数组和目标数组</span><br><span class="line">model.fit([headline_data, additional_data], [labels, labels],</span><br><span class="line">          epochs=50, batch_size=32)</span><br><span class="line"></span><br><span class="line"># 根据 name 参数编译</span><br><span class="line">model.compile(optimizer=&apos;rmsprop&apos;,</span><br><span class="line">              loss=&#123;&apos;main_output&apos;: &apos;binary_crossentropy&apos;, &apos;aux_output&apos;: &apos;binary_crossentropy&apos;&#125;,</span><br><span class="line">              loss_weights=&#123;&apos;main_output&apos;: 1., &apos;aux_output&apos;: 0.2&#125;)</span><br><span class="line"></span><br><span class="line"># 然后使用以下方式训练：</span><br><span class="line">model.fit(&#123;&apos;main_input&apos;: headline_data, &apos;aux_input&apos;: additional_data&#125;,</span><br><span class="line">          &#123;&apos;main_output&apos;: labels, &apos;aux_output&apos;: labels&#125;,</span><br><span class="line">          epochs=50, batch_size=32)</span><br></pre></td></tr></table></figure></li><li><p><a href="https://zhuanlan.zhihu.com/p/65192903" target="_blank" rel="noopener">tokenizer与embedding</a></p></li><li><p>GELU</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">from keras.layers import Activation</span><br><span class="line">from keras.utils.generic_utils import get_custom_objects</span><br><span class="line"></span><br><span class="line">def custom_gelu(x):</span><br><span class="line">    return 0.5 * x * (1 + tf.tanh(tf.sqrt(2 / np.pi) * (x + 0.044715 * tf.pow(x, 3))))</span><br><span class="line">get_custom_objects().update(&#123;&apos;custom_gelu&apos;: Activation(custom_gelu)&#125;)</span><br><span class="line">fit1.add(Dense(output_dim=1, activation=custom_gelu))</span><br></pre></td></tr></table></figure></li><li><p><a href="https://keras.io/zh/layers/recurrent/" target="_blank" rel="noopener">RNN (LSTM, GRU) 模型</a></p><ul><li><code>lstm1, lstm_h, lstm_c = LSTM(hideen_size, return_sequences=True, return_state=True)(input)</code><br>返回lstm的每层隐状态<code>lstm1</code>,最后输出<code>lstm_h</code>,最后的单元状态<code>lstm_c</code>。</li></ul></li><li><p><a href="https://keras.io/api/layers/recurrent_layers/bidirectional/" target="_blank" rel="noopener">Bidirenctial layer</a></p><ul><li><code>lstm_out = Bidirectional(LSTM(10, return_sequences=True)(input))</code></li><li><p>也可以分开写</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">forward_layer = LSTM(10, return_sequences=True)(input)</span><br><span class="line">backward_layer = LSTM(10, activation=&apos;relu&apos;, return_sequences=True,</span><br><span class="line">                       go_backwards=True)(input)</span><br></pre></td></tr></table></figure></li><li><p><a href="https://stackoverflow.com/questions/47923370/keras-bidirectional-lstm-seq2seq" target="_blank" rel="noopener">使用<code>return_state</code></a></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">encoder_inputs = Input(shape=(None, num_encoder_tokens))</span><br><span class="line">encoder = Bidirectional(LSTM(latent_dim, return_state=True))</span><br><span class="line">encoder_outputs, forward_h, forward_c, backward_h, backward_c = encoder(encoder_inputs)</span><br><span class="line"></span><br><span class="line">state_h = Concatenate()([forward_h, backward_h])</span><br><span class="line">state_c = Concatenate()([forward_c, backward_c])</span><br><span class="line"></span><br><span class="line">encoder_states = [state_h, state_c]</span><br><span class="line"></span><br><span class="line">decoder_inputs = Input(shape=(None, num_decoder_tokens))</span><br><span class="line">decoder_lstm = LSTM(latent_dim * 2, return_sequences=True, return_state=True)</span><br><span class="line">decoder_outputs, _, _ = decoder_lstm(decoder_inputs, initial_state=encoder_states)</span><br></pre></td></tr></table></figure></li></ul></li><li><p><a href="https://www.sohu.com/a/407812302_500659" target="_blank" rel="noopener">mask 使用方法</a></p><ul><li><p>Masking layer:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">mask = keras.layers.Masking(mask_value= 0, input_shape=(time_step,feature_size))(input)</span><br><span class="line">lstm_output = keras.layers.LSTM(hidden_size, return_sequences= True)(mask)</span><br></pre></td></tr></table></figure></li><li><p>Embedding layer:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">embed = keras.layers.Embedding(vocab_size, embedding_size, mask_zero= True)(input)</span><br><span class="line">lstm_output = keras.layers.LSTM(hidden_size, return_sequences= True)(emded)</span><br></pre></td></tr></table></figure></li></ul></li><li><p><a href="https://keras.io/zh/optimizers/" target="_blank" rel="noopener">optimizer</a></p><ul><li><code>tf.keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)</code><ul><li><code>decay</code>: 学习率衰减</li></ul></li><li><p>Learning rate</p><ul><li><p>tf 2.0: <code>tf.keras.optimizers.schedules.LearningRateSchedule</code></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):</span><br><span class="line">    def __init__(self, d_model, warmup_steps=4000):</span><br><span class="line">        super(CustomSchedule, self).__init__()</span><br><span class="line"></span><br><span class="line">        self.d_model = d_model</span><br><span class="line">        self.d_model = tf.cast(self.d_model, tf.float32)</span><br><span class="line"></span><br><span class="line">        self.warmup_steps = warmup_steps</span><br><span class="line"></span><br><span class="line">    def __call__(self, step):</span><br><span class="line">        arg1 = tf.math.rsqrt(step)</span><br><span class="line">        arg2 = step * (self.warmup_steps ** -1.5)</span><br><span class="line"></span><br><span class="line">        return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)</span><br><span class="line"></span><br><span class="line">learning_rate = CustomSchedule(200)</span><br><span class="line">custom_adam = tf.keras.optimizers.Adam(learning_rate, beta_1=0.9, beta_2=0.98, </span><br><span class="line">                                 epsilon=1e-9)</span><br></pre></td></tr></table></figure></li><li><p>keras: <code>keras.callbacks.LearningRateScheduler(schedule)</code></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">import keras.backend as K</span><br><span class="line">from keras.callbacks import LearningRateScheduler</span><br><span class="line"> </span><br><span class="line">def scheduler(epoch):</span><br><span class="line">    # 每隔100个epoch，学习率减小为原来的1/10</span><br><span class="line">    if epoch % 100 == 0 and epoch != 0:</span><br><span class="line">        lr = K.get_value(model.optimizer.lr)</span><br><span class="line">        K.set_value(model.optimizer.lr, lr * 0.1)</span><br><span class="line">        print(&quot;lr changed to &#123;&#125;&quot;.format(lr * 0.1))</span><br><span class="line">    return K.get_value(model.optimizer.lr)</span><br><span class="line"> </span><br><span class="line">reduce_lr = LearningRateScheduler(scheduler)</span><br><span class="line">model.fit(train_x, train_y, batch_size=32, epochs=300, callbacks=[reduce_lr])</span><br></pre></td></tr></table></figure></li></ul></li><li><p><a href="https://blog.csdn.net/zzc15806/article/details/79711114" target="_blank" rel="noopener">Reduce LR On Plateau</a><br><code>keras.callbacks.ReduceLROnPlateau(monitor=&#39;val_loss&#39;, factor=0.1, patience=10, verbose=0, mode=&#39;auto&#39;, epsilon=0.0001, cooldown=0, min_lr=0)</code></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">from keras.callbacks import ReduceLROnPlateau</span><br><span class="line">reduce_lr = ReduceLROnPlateau(monitor=&apos;val_loss&apos;, patience=10, mode=&apos;auto&apos;)</span><br><span class="line">model.fit(train_x, train_y, batch_size=32, epochs=300, validation_split=0.1, callbacks=[reduce_lr])</span><br></pre></td></tr></table></figure></li></ul></li></ol>]]></content>
      
      
      <categories>
          
          <category> Tensorflow </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Tensorflow </tag>
            
            <tag> Keras </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>剑指offer</title>
      <link href="/2020/04/26/offer/"/>
      <url>/2020/04/26/offer/</url>
      
        <content type="html"><![CDATA[<h1 id="数组"><a href="#数组" class="headerlink" title="数组"></a>数组</h1><ol><li>数组占据一块连续的内存并且按照顺序存储，空间效率低</li><li>哈希表：数组的下标设为哈希表的键值，数组中的数字为哈希表的值，“键值-值”的配对</li><li>数组中重复的数字<ul><li>set</li><li>list</li></ul></li><li>二维数组中的查找</li></ol><h1 id="字符串"><a href="#字符串" class="headerlink" title="字符串"></a>字符串</h1><ol><li>替换空格<ul><li><code>s.replace(&#39; &#39;,&#39;%20&#39;)</code></li></ul></li></ol><h1 id="链表"><a href="#链表" class="headerlink" title="链表"></a>链表</h1><ol><li>链表是一种动态数据结构</li><li><p>定义：</p> <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">class ListNode:</span><br><span class="line">    def __init__(self, x):</span><br><span class="line">        self.val = x</span><br><span class="line">        self.next = None</span><br></pre></td></tr></table></figure></li><li><p>从尾到头打印链表</p> <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">class Solution:</span><br><span class="line">   # 返回从尾部到头部的列表值序列，例如[1,2,3]</span><br><span class="line">    def printListFromTailToHead(self, listNode):</span><br><span class="line">        # write code here</span><br><span class="line">        l = []</span><br><span class="line">        head = listNode</span><br><span class="line">        while head:</span><br><span class="line">            l.insert(0, head.val)</span><br><span class="line">            head = head.next</span><br><span class="line">        return l</span><br></pre></td></tr></table></figure></li></ol><h1 id="树"><a href="#树" class="headerlink" title="树"></a>树</h1><ol><li><p>二叉树</p><ul><li>前序遍历： 根-左-右</li><li>中序遍历： 左-根-右</li><li>后序遍历： 左-右-根</li><li>定义：<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">class TreeNode:</span><br><span class="line">    def __init__(self, x):</span><br><span class="line">        self.val = x</span><br><span class="line">        self.left = None</span><br><span class="line">        self.right = None</span><br></pre></td></tr></table></figure></li></ul></li><li><p>重建二叉树</p><ul><li>前序遍历的第一个数字尾根节点，中序遍历中，根节点前的为左子树节点，后面为右子树</li><li>可以用递归重建：<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">class Solution:</span><br><span class="line">    # 返回构造的TreeNode根节点</span><br><span class="line">    def reConstructBinaryTree(self, pre, tin):</span><br><span class="line">        # write code here</span><br><span class="line">        if len(pre)==0:</span><br><span class="line">            return None</span><br><span class="line">        if len(pre)==1:</span><br><span class="line">            return TreeNode(pre[0])</span><br><span class="line">        else:</span><br><span class="line">            root = TreeNode(pre[0])</span><br><span class="line">            r_idx = tin.index(pre[0])</span><br><span class="line">            root.left = self.reConstructBinaryTree(pre[1:r_idx+1],tin[0:r_idx])</span><br><span class="line">            root.right = self.reConstructBinaryTree(pre[r_idx+1:],tin[r_idx+1:])</span><br><span class="line">            return root</span><br></pre></td></tr></table></figure></li></ul></li><li><p>二叉树的下一个节点</p></li></ol><h1 id="栈和队列"><a href="#栈和队列" class="headerlink" title="栈和队列"></a>栈和队列</h1><ol><li>栈：后进后出，最后被压入(push)栈的元素会第一个被弹出(pop)</li><li>队列：先进先出，第一个进入队列的第一个出来。</li><li>用两个栈实现队列：先将元素插入stack1,再将stack1中元素倒序插入stack2，从stack2中出栈： <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">class Solution:</span><br><span class="line">    def __init__(self):</span><br><span class="line">        self.stack1=[]</span><br><span class="line">        self.stack2=[]</span><br><span class="line">    def push(self, node):</span><br><span class="line">        # write code here</span><br><span class="line">        self.stack1.append(node)</span><br><span class="line">    def pop(self):</span><br><span class="line">        # return xx</span><br><span class="line">        if self.stack2==[]:</span><br><span class="line">            while self.stack1:</span><br><span class="line">                self.stack2.append(self.stack1.pop())</span><br><span class="line">        return self.stack2.pop()</span><br></pre></td></tr></table></figure></li></ol><h1 id="递归和循环"><a href="#递归和循环" class="headerlink" title="递归和循环"></a>递归和循环</h1><ol><li>递归：在一个函数内部调用函数自身</li><li>循环：重复运算</li></ol>]]></content>
      
      
      <categories>
          
          <category> Python </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Python </tag>
            
            <tag> Data Structure </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Matplotlib绘图</title>
      <link href="/2020/04/23/matplotlib/"/>
      <url>/2020/04/23/matplotlib/</url>
      
        <content type="html"><![CDATA[<p>主要参考：<a href="https://morvanzhou.github.io/tutorials/data-manipulation/plt/" target="_blank" rel="noopener">Matplotlib Python3 数据可视化神器</a></p><ol><li><p>基础用法</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">import matplotlib.pyplot as plt</span><br><span class="line">import numpy as np</span><br><span class="line"></span><br><span class="line"># 使用np.linspace定义x：范围是(-1,1);个数是50. 仿真一维数据组(x ,y)表示曲线1.</span><br><span class="line">x = np.linspace(-1, 1, 50)</span><br><span class="line">y = 2*x + 1</span><br><span class="line"></span><br><span class="line"># 使用plt.figure定义一个图像窗口. 使用plt.plot画(x ,y)曲线. 使用plt.show显示图像.</span><br><span class="line">plt.figure()</span><br><span class="line">plt.plot(x, y)</span><br><span class="line">plt.show()</span><br></pre></td></tr></table></figure></li><li><p>画多根线条</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"># 定义曲线2</span><br><span class="line">y2 = x**2</span><br><span class="line"></span><br><span class="line"># 绘制两条曲线</span><br><span class="line">plt.figure(num=3, figsize=(8, 5),)</span><br><span class="line">plt.plot(x, y2)</span><br><span class="line">plt.plot(x, y, color=&apos;red&apos;, linewidth=1.0, linestyle=&apos;--&apos;)</span><br><span class="line">plt.show()</span><br></pre></td></tr></table></figure></li><li><p>调整名字和间距<br>使用plt.xlim设置x坐标轴范围：(-1, 2)； 使用plt.ylim设置y坐标轴范围：(-2, 3)； 使用plt.xlabel设置x坐标轴名称：’I am x’； 使用plt.ylabel设置y坐标轴名称：’I am y’：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">plt.xlim((-1, 2))</span><br><span class="line">plt.ylim((-2, 3))</span><br><span class="line">plt.xlabel(&apos;I am x&apos;)</span><br><span class="line">plt.ylabel(&apos;I am y&apos;)</span><br></pre></td></tr></table></figure></li></ol><p>添加在<code>plt.show()</code>前。</p><p>使用np.linspace定义范围以及个数：范围是(-1,2);个数是5. 使用print打印出新定义的范围. 使用plt.xticks设置x轴刻度：范围是(-1,2);个数是5.<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">new_ticks = np.linspace(-1, 2, 5)</span><br><span class="line">print(new_ticks)</span><br><span class="line">plt.xticks(new_ticks)</span><br></pre></td></tr></table></figure></p><p>使用plt.yticks设置y轴刻度以及名称：刻度为[-2, -1.8, -1, 1.22, 3]；对应刻度的名称为[‘really bad’,’bad’,’normal’,’good’, ‘really good’]. 使用plt.show显示图像.<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">plt.yticks([-2, -1.8, -1, 1.22, 3],[r&apos;$really\ bad$&apos;, r&apos;$bad$&apos;, r&apos;$normal$&apos;, r&apos;$good$&apos;, r&apos;$really\ good$&apos;])</span><br><span class="line">plt.show()</span><br></pre></td></tr></table></figure></p><p>使用plt.gca获取当前坐标轴信息. 使用.spines设置边框：右侧边框；使用.set_color设置边框颜色：默认白色； 使用.spines设置边框：上边框；使用.set_color设置边框颜色：默认白色；<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">ax = plt.gca()</span><br><span class="line">ax.spines[&apos;right&apos;].set_color(&apos;none&apos;)</span><br><span class="line">ax.spines[&apos;top&apos;].set_color(&apos;none&apos;)</span><br><span class="line">plt.show()</span><br></pre></td></tr></table></figure></p>]]></content>
      
      
      <categories>
          
          <category> Python </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Python </tag>
            
            <tag> Matplotlib </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>AI for Medicine</title>
      <link href="/2020/04/22/ai4md/"/>
      <url>/2020/04/22/ai4md/</url>
      
        <content type="html"><![CDATA[<p>最近COURSERA上新开了一门课:<a href="https://www.coursera.org/specializations/ai-for-medicine" target="_blank" rel="noopener">AI for Medicine Specialization</a></p><h1 id="AI-for-Medical-Diagnosis"><a href="#AI-for-Medical-Diagnosis" class="headerlink" title="AI for Medical Diagnosis"></a><a href="https://www.coursera.org/learn/ai-for-medical-diagnosis/home/welcome" target="_blank" rel="noopener">AI for Medical Diagnosis</a></h1><ol><li><p><a href="https://arxiv.org/abs/1705.02315" target="_blank" rel="noopener">ChestX-ray8 dataset</a></p></li><li><p>Image Preprocessing in Keras</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br></pre></td><td class="code"><pre><span class="line"># Import data generator from keras</span><br><span class="line">from keras.preprocessing.image import ImageDataGenerator</span><br><span class="line"></span><br><span class="line"># Normalize images</span><br><span class="line">image_generator = ImageDataGenerator(</span><br><span class="line">    samplewise_center=True, #Set each sample mean to 0.</span><br><span class="line">    samplewise_std_normalization= True # Divide each input by its standard deviation</span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"># Flow from directory with specified batch size and target image size</span><br><span class="line">generator = image_generator.flow_from_dataframe(</span><br><span class="line">        dataframe=train_df,</span><br><span class="line">        directory=&quot;nih/images-small/&quot;,</span><br><span class="line">        x_col=&quot;Image&quot;, # features</span><br><span class="line">        y_col= [&apos;Mass&apos;], # labels</span><br><span class="line">        class_mode=&quot;raw&quot;, # &apos;Mass&apos; column should be in train_df</span><br><span class="line">        batch_size= 1, # images per batch</span><br><span class="line">        shuffle=False, # shuffle the rows or not</span><br><span class="line">        target_size=(320,320) # width and height of output image</span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"># Plot a processed image</span><br><span class="line">sns.set_style(&quot;white&quot;)</span><br><span class="line">generated_image, label = generator.__getitem__(0)</span><br><span class="line">plt.imshow(generated_image[0], cmap=&apos;gray&apos;)</span><br><span class="line">plt.colorbar()</span><br><span class="line">plt.title(&apos;Raw Chest X Ray Image&apos;)</span><br><span class="line">print(f&quot;The dimensions of the image are &#123;generated_image.shape[1]&#125; pixels width and &#123;generated_image.shape[2]&#125; pixels height&quot;)</span><br><span class="line">print(f&quot;The maximum pixel value is &#123;generated_image.max():.4f&#125; and the minimum is &#123;generated_image.min():.4f&#125;&quot;)</span><br><span class="line">print(f&quot;The mean value of the pixels is &#123;generated_image.mean():.4f&#125; and the standard deviation is &#123;generated_image.std():.4f&#125;&quot;)</span><br><span class="line"></span><br><span class="line"># Include a histogram of the distribution of the pixels</span><br><span class="line">sns.set()</span><br><span class="line">plt.figure(figsize=(10, 7))</span><br><span class="line"></span><br><span class="line"># Plot histogram for original iamge</span><br><span class="line">sns.distplot(raw_image.ravel(), </span><br><span class="line">             label=f&apos;Original Image: mean &#123;np.mean(raw_image):.4f&#125; - Standard Deviation &#123;np.std(raw_image):.4f&#125; \n &apos;</span><br><span class="line">             f&apos;Min pixel value &#123;np.min(raw_image):.4&#125; - Max pixel value &#123;np.max(raw_image):.4&#125;&apos;,</span><br><span class="line">             color=&apos;blue&apos;, </span><br><span class="line">             kde=False)</span><br><span class="line"></span><br><span class="line"># Plot histogram for generated image</span><br><span class="line">sns.distplot(generated_image[0].ravel(), </span><br><span class="line">             label=f&apos;Generated Image: mean &#123;np.mean(generated_image[0]):.4f&#125; - Standard Deviation &#123;np.std(generated_image[0]):.4f&#125; \n&apos;</span><br><span class="line">             f&apos;Min pixel value &#123;np.min(generated_image[0]):.4&#125; - Max pixel value &#123;np.max(generated_image[0]):.4&#125;&apos;, </span><br><span class="line">             color=&apos;red&apos;, </span><br><span class="line">             kde=False)</span><br><span class="line"></span><br><span class="line"># Place legends</span><br><span class="line">plt.legend()</span><br><span class="line">plt.title(&apos;Distribution of Pixel Intensities in the Image&apos;)</span><br><span class="line">plt.xlabel(&apos;Pixel Intensity&apos;)</span><br><span class="line">plt.ylabel(&apos;# Pixel&apos;)</span><br></pre></td></tr></table></figure></li><li><p>Dense net</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br></pre></td><td class="code"><pre><span class="line"># Import Densenet from Keras</span><br><span class="line">from keras.applications.densenet import DenseNet121</span><br><span class="line">from keras.layers import Dense, GlobalAveragePooling2D</span><br><span class="line">from keras.models import Model</span><br><span class="line">from keras import backend as K</span><br><span class="line"></span><br><span class="line"># Create the base pre-trained model</span><br><span class="line">base_model = DenseNet121(weights=&apos;./nih/densenet.hdf5&apos;, include_top=False);</span><br><span class="line"></span><br><span class="line"># Print the model summary</span><br><span class="line">base_model.summary()</span><br><span class="line"></span><br><span class="line"># The number of output channels</span><br><span class="line">print(&quot;The output has 1024 channels&quot;)</span><br><span class="line">x = base_model.output</span><br><span class="line"></span><br><span class="line"># Add a global spatial average pooling layer</span><br><span class="line">x_pool = GlobalAveragePooling2D()(x)</span><br><span class="line"></span><br><span class="line"># Define a set of five class labels to use as an example</span><br><span class="line">labels = [&apos;Emphysema&apos;, </span><br><span class="line">          &apos;Hernia&apos;, </span><br><span class="line">          &apos;Mass&apos;, </span><br><span class="line">          &apos;Pneumonia&apos;,  </span><br><span class="line">          &apos;Edema&apos;]</span><br><span class="line">n_classes = len(labels)</span><br><span class="line">print(f&quot;In this example, you want your model to identify &#123;n_classes&#125; classes&quot;)</span><br><span class="line"></span><br><span class="line"># Add a logistic layer the same size as the number of classes you&apos;re trying to predict</span><br><span class="line">predictions = Dense(n_classes, activation=&quot;sigmoid&quot;)(x_pool)</span><br><span class="line">print(f&quot;Predictions have &#123;n_classes&#125; units, one for each class&quot;)</span><br><span class="line"></span><br><span class="line"># Create an updated model</span><br><span class="line">model = Model(inputs=base_model.input, outputs=predictions)</span><br><span class="line"></span><br><span class="line"># Compile the model</span><br><span class="line">model.compile(optimizer=&apos;adam&apos;,</span><br><span class="line">              loss=&apos;categorical_crossentropy&apos;)</span><br><span class="line"># (You&apos;ll customize the loss function in the assignment!)</span><br></pre></td></tr></table></figure></li></ol><h1 id="Chest-X-Ray-Medical-Diagnosis-with-Deep-Learning"><a href="#Chest-X-Ray-Medical-Diagnosis-with-Deep-Learning" class="headerlink" title="Chest X-Ray Medical Diagnosis with Deep Learning"></a>Chest X-Ray Medical Diagnosis with Deep Learning</h1><ol><li><p>Import packages and functions</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">import numpy as np</span><br><span class="line">import pandas as pd</span><br><span class="line">import seaborn as sns</span><br><span class="line">import matplotlib.pyplot as plt</span><br><span class="line"></span><br><span class="line">from keras.preprocessing.image import ImageDataGenerator</span><br><span class="line">from keras.applications.densenet import DenseNet121</span><br><span class="line">from keras.layers import Dense, GlobalAveragePooling2D</span><br><span class="line">from keras.models import Model</span><br><span class="line">from keras import backend as K</span><br><span class="line"></span><br><span class="line">from keras.models import load_model</span><br><span class="line"></span><br><span class="line">import util</span><br></pre></td></tr></table></figure></li><li><p>Load the Datasets</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">train_df = pd.read_csv(&quot;nih/train-small.csv&quot;)</span><br><span class="line">valid_df = pd.read_csv(&quot;nih/valid-small.csv&quot;)</span><br><span class="line">test_df = pd.read_csv(&quot;nih/test.csv&quot;)</span><br></pre></td></tr></table></figure><ul><li><p>Preventing Data Leakage</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><span class="line"># UNQ_C1 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)</span><br><span class="line">def check_for_leakage(df1, df2, patient_col):</span><br><span class="line">    &quot;&quot;&quot;</span><br><span class="line">    Return True if there any patients are in both df1 and df2.</span><br><span class="line"></span><br><span class="line">    Args:</span><br><span class="line">        df1 (dataframe): dataframe describing first dataset</span><br><span class="line">        df2 (dataframe): dataframe describing second dataset</span><br><span class="line">        patient_col (str): string name of column with patient IDs</span><br><span class="line">    </span><br><span class="line">    Returns:</span><br><span class="line">        leakage (bool): True if there is leakage, otherwise False</span><br><span class="line">    &quot;&quot;&quot;</span><br><span class="line"></span><br><span class="line">    ### START CODE HERE (REPLACE INSTANCES OF &apos;None&apos; with your code) ###</span><br><span class="line">    </span><br><span class="line">    df1_patients_unique = list(set(df1[patient_col].tolist()))</span><br><span class="line">    df2_patients_unique = list(set(df2[patient_col].tolist()))</span><br><span class="line">    </span><br><span class="line">    patients_in_both_groups = list(set(df1_patients_unique+df2_patients_unique))</span><br><span class="line"></span><br><span class="line">    # leakage contains true if there is patient overlap, otherwise false.</span><br><span class="line">    leakage = False # boolean (true if there is at least 1 patient in both groups)</span><br><span class="line">    if len(patients_in_both_groups)&lt;len(df1_patients_unique)+len(df2_patients_unique):</span><br><span class="line">        leakage = True</span><br><span class="line">    </span><br><span class="line">    ### END CODE HERE ###</span><br><span class="line">    </span><br><span class="line">    return leakage</span><br></pre></td></tr></table></figure></li><li><p>Preparing Images</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br></pre></td><td class="code"><pre><span class="line">def get_train_generator(df, image_dir, x_col, y_cols, shuffle=True, batch_size=8, seed=1, target_w = 320, target_h = 320):</span><br><span class="line">  &quot;&quot;&quot;</span><br><span class="line">  Return generator for training set, normalizing using batch</span><br><span class="line">  statistics.</span><br><span class="line"></span><br><span class="line">  Args:</span><br><span class="line">    train_df (dataframe): dataframe specifying training data.</span><br><span class="line">    image_dir (str): directory where image files are held.</span><br><span class="line">    x_col (str): name of column in df that holds filenames.</span><br><span class="line">    y_cols (list): list of strings that hold y labels for images.</span><br><span class="line">    sample_size (int): size of sample to use for normalization statistics.</span><br><span class="line">    batch_size (int): images per batch to be fed into model during training.</span><br><span class="line">    seed (int): random seed.</span><br><span class="line">    target_w (int): final width of input images.</span><br><span class="line">    target_h (int): final height of input images.</span><br><span class="line">  </span><br><span class="line">  Returns:</span><br><span class="line">      train_generator (DataFrameIterator): iterator over training set</span><br><span class="line">  &quot;&quot;&quot;        </span><br><span class="line">  print(&quot;getting train generator...&quot;) </span><br><span class="line">  # normalize images</span><br><span class="line">  image_generator = ImageDataGenerator(</span><br><span class="line">      samplewise_center=True,</span><br><span class="line">      samplewise_std_normalization= True)</span><br><span class="line">  </span><br><span class="line">  # flow from directory with specified batch size</span><br><span class="line">  # and target image size</span><br><span class="line">  generator = image_generator.flow_from_dataframe(</span><br><span class="line">          dataframe=df,</span><br><span class="line">          directory=image_dir,</span><br><span class="line">          x_col=x_col,</span><br><span class="line">          y_col=y_cols,</span><br><span class="line">          class_mode=&quot;raw&quot;,</span><br><span class="line">          batch_size=batch_size,</span><br><span class="line">          shuffle=shuffle,</span><br><span class="line">          seed=seed,</span><br><span class="line">          target_size=(target_w,target_h))</span><br><span class="line">  </span><br><span class="line">  return generator</span><br><span class="line">  </span><br><span class="line">def get_test_and_valid_generator(valid_df, test_df, train_df, image_dir, x_col, y_cols, sample_size=100, batch_size=8, seed=1, target_w = 320, target_h = 320):</span><br><span class="line">  &quot;&quot;&quot;</span><br><span class="line">  Return generator for validation set and test test set using </span><br><span class="line">  normalization statistics from training set.</span><br><span class="line"></span><br><span class="line">  Args:</span><br><span class="line">    valid_df (dataframe): dataframe specifying validation data.</span><br><span class="line">    test_df (dataframe): dataframe specifying test data.</span><br><span class="line">    train_df (dataframe): dataframe specifying training data.</span><br><span class="line">    image_dir (str): directory where image files are held.</span><br><span class="line">    x_col (str): name of column in df that holds filenames.</span><br><span class="line">    y_cols (list): list of strings that hold y labels for images.</span><br><span class="line">    sample_size (int): size of sample to use for normalization statistics.</span><br><span class="line">    batch_size (int): images per batch to be fed into model during training.</span><br><span class="line">    seed (int): random seed.</span><br><span class="line">    target_w (int): final width of input images.</span><br><span class="line">    target_h (int): final height of input images.</span><br><span class="line">  </span><br><span class="line">  Returns:</span><br><span class="line">      test_generator (DataFrameIterator) and valid_generator: iterators over test set and validation set respectively</span><br><span class="line">  &quot;&quot;&quot;</span><br><span class="line">  print(&quot;getting train and valid generators...&quot;)</span><br><span class="line">  # get generator to sample dataset</span><br><span class="line">  raw_train_generator = ImageDataGenerator().flow_from_dataframe(</span><br><span class="line">      dataframe=train_df, </span><br><span class="line">      directory=IMAGE_DIR, </span><br><span class="line">      x_col=&quot;Image&quot;, </span><br><span class="line">      y_col=labels, </span><br><span class="line">      class_mode=&quot;raw&quot;, </span><br><span class="line">      batch_size=sample_size, </span><br><span class="line">      shuffle=True, </span><br><span class="line">      target_size=(target_w, target_h))</span><br><span class="line">  </span><br><span class="line">  # get data sample</span><br><span class="line">  batch = raw_train_generator.next()</span><br><span class="line">  data_sample = batch[0]</span><br><span class="line"></span><br><span class="line">  # use sample to fit mean and std for test set generator</span><br><span class="line">  image_generator = ImageDataGenerator(</span><br><span class="line">      featurewise_center=True,</span><br><span class="line">      featurewise_std_normalization= True)</span><br><span class="line">  </span><br><span class="line">  # fit generator to sample from training data</span><br><span class="line">  image_generator.fit(data_sample)</span><br><span class="line"></span><br><span class="line">  # get test generator</span><br><span class="line">  valid_generator = image_generator.flow_from_dataframe(</span><br><span class="line">          dataframe=valid_df,</span><br><span class="line">          directory=image_dir,</span><br><span class="line">          x_col=x_col,</span><br><span class="line">          y_col=y_cols,</span><br><span class="line">          class_mode=&quot;raw&quot;,</span><br><span class="line">          batch_size=batch_size,</span><br><span class="line">          shuffle=False,</span><br><span class="line">          seed=seed,</span><br><span class="line">          target_size=(target_w,target_h))</span><br><span class="line"></span><br><span class="line">  test_generator = image_generator.flow_from_dataframe(</span><br><span class="line">          dataframe=test_df,</span><br><span class="line">          directory=image_dir,</span><br><span class="line">          x_col=x_col,</span><br><span class="line">          y_col=y_cols,</span><br><span class="line">          class_mode=&quot;raw&quot;,</span><br><span class="line">          batch_size=batch_size,</span><br><span class="line">          shuffle=False,</span><br><span class="line">          seed=seed,</span><br><span class="line">          target_size=(target_w,target_h))</span><br><span class="line">  return valid_generator, test_generator</span><br><span class="line"></span><br><span class="line">IMAGE_DIR = &quot;nih/images-small/&quot;</span><br><span class="line">train_generator = get_train_generator(train_df, IMAGE_DIR, &quot;Image&quot;, labels)</span><br><span class="line">valid_generator, test_generator= get_test_and_valid_generator(valid_df, test_df, train_df, IMAGE_DIR, &quot;Image&quot;, labels)</span><br><span class="line"></span><br><span class="line">x, y = train_generator.__getitem__(0)</span><br><span class="line">plt.imshow(x[0]);</span><br></pre></td></tr></table></figure></li></ul></li><li><p>Model Development</p><ul><li><p>Addressing class imbalance</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br></pre></td><td class="code"><pre><span class="line"># UNQ_C2 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)</span><br><span class="line">def compute_class_freqs(labels):</span><br><span class="line">    &quot;&quot;&quot;</span><br><span class="line">    Compute positive and negative frequences for each class.</span><br><span class="line"></span><br><span class="line">    Args:</span><br><span class="line">        labels (np.array): matrix of labels, size (num_examples, num_classes)</span><br><span class="line">    Returns:</span><br><span class="line">        positive_frequencies (np.array): array of positive frequences for each</span><br><span class="line">                                         class, size (num_classes)</span><br><span class="line">        negative_frequencies (np.array): array of negative frequences for each</span><br><span class="line">                                         class, size (num_classes)</span><br><span class="line">    &quot;&quot;&quot;</span><br><span class="line">    ### START CODE HERE (REPLACE INSTANCES OF &apos;None&apos; with your code) ###</span><br><span class="line">    </span><br><span class="line">    # total number of patients (rows)</span><br><span class="line">    N = len(labels)</span><br><span class="line">    </span><br><span class="line">    positive_frequencies = sum(labels)/N</span><br><span class="line">    negative_frequencies = 1-positive_frequencies</span><br><span class="line"></span><br><span class="line">    ### END CODE HERE ###</span><br><span class="line">    return positive_frequencies, negative_frequencies</span><br><span class="line"></span><br><span class="line">freq_pos, freq_neg = compute_class_freqs(train_generator.labels)</span><br><span class="line">pos_weights = freq_neg</span><br><span class="line">neg_weights = freq_pos</span><br><span class="line">pos_contribution = freq_pos * pos_weights </span><br><span class="line">neg_contribution = freq_neg * neg_weights</span><br><span class="line"></span><br><span class="line"># UNQ_C3 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)</span><br><span class="line">def get_weighted_loss(pos_weights, neg_weights, epsilon=1e-7):</span><br><span class="line">    &quot;&quot;&quot;</span><br><span class="line">    Return weighted loss function given negative weights and positive weights.</span><br><span class="line"></span><br><span class="line">    Args:</span><br><span class="line">      pos_weights (np.array): array of positive weights for each class, size (num_classes)</span><br><span class="line">      neg_weights (np.array): array of negative weights for each class, size (num_classes)</span><br><span class="line">    </span><br><span class="line">    Returns:</span><br><span class="line">      weighted_loss (function): weighted loss function</span><br><span class="line">    &quot;&quot;&quot;</span><br><span class="line">    def weighted_loss(y_true, y_pred):</span><br><span class="line">        &quot;&quot;&quot;</span><br><span class="line">        Return weighted loss value. </span><br><span class="line"></span><br><span class="line">        Args:</span><br><span class="line">            y_true (Tensor): Tensor of true labels, size is (num_examples, num_classes)</span><br><span class="line">            y_pred (Tensor): Tensor of predicted labels, size is (num_examples, num_classes)</span><br><span class="line">        Returns:</span><br><span class="line">            loss (Tensor): overall scalar loss summed across all classes</span><br><span class="line">        &quot;&quot;&quot;</span><br><span class="line">        # initialize loss to zero</span><br><span class="line">        loss = 0.0</span><br><span class="line">        </span><br><span class="line">        ### START CODE HERE (REPLACE INSTANCES OF &apos;None&apos; with your code) ###</span><br><span class="line"></span><br><span class="line">        for i in range(len(pos_weights)):</span><br><span class="line">            # for each class, add average weighted loss for that class </span><br><span class="line">            loss += K.mean(-(pos_weights[i]*y_true[:,i]*K.log(y_pred[:,i]+epsilon)+neg_weights[i]*(1-y_true[:,i])*K.log(1-y_pred[:,i]+epsilon)))</span><br><span class="line">        return loss</span><br><span class="line">    </span><br><span class="line">        ### END CODE HERE ###</span><br><span class="line">    return weighted_loss</span><br></pre></td></tr></table></figure></li><li><p>DenseNet121</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"># create the base pre-trained model</span><br><span class="line">base_model = DenseNet121(weights=&apos;./nih/densenet.hdf5&apos;, include_top=False)</span><br><span class="line"></span><br><span class="line">x = base_model.output</span><br><span class="line"></span><br><span class="line"># add a global spatial average pooling layer</span><br><span class="line">x = GlobalAveragePooling2D()(x)</span><br><span class="line"></span><br><span class="line"># and a logistic layer</span><br><span class="line">predictions = Dense(len(labels), activation=&quot;sigmoid&quot;)(x)</span><br><span class="line"></span><br><span class="line">model = Model(inputs=base_model.input, outputs=predictions)</span><br><span class="line">model.compile(optimizer=&apos;adam&apos;, loss=get_weighted_loss(pos_weights, neg_weights))</span><br></pre></td></tr></table></figure></li></ul></li><li><p>Training</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">history = model.fit_generator(train_generator, </span><br><span class="line">                              validation_data=valid_generator,</span><br><span class="line">                              steps_per_epoch=100, </span><br><span class="line">                              validation_steps=25, </span><br><span class="line">                              epochs = 3)</span><br><span class="line"></span><br><span class="line">plt.plot(history.history[&apos;loss&apos;])</span><br><span class="line">plt.ylabel(&quot;loss&quot;)</span><br><span class="line">plt.xlabel(&quot;epoch&quot;)</span><br><span class="line">plt.title(&quot;Training Loss Curve&quot;)</span><br><span class="line">plt.show()</span><br></pre></td></tr></table></figure><ul><li><code>model.load_weights(&quot;./nih/pretrained_model.h5&quot;)</code></li></ul></li><li>Prediction and Evaluation<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">predicted_vals = model.predict_generator(test_generator, steps = len(test_generator))</span><br><span class="line">auc_rocs = util.get_roc_curve(labels, predicted_vals, test_generator)</span><br></pre></td></tr></table></figure></li></ol><h1 id="ACCURACY"><a href="#ACCURACY" class="headerlink" title="ACCURACY"></a>ACCURACY</h1><ol><li><code>Accuracy = P(+|disease)P(disease)+P(-|normal)P(normal)</code><ul><li>P(+|disease): Sensitivity(true positive rate)</li><li>P(-|normal): Specificity(true negtive rate)</li><li>P(disease): Prevalence</li><li><code>Accuracy = Sensitivity*Prevalence+Specificity*(1-Prevalence)</code></li></ul></li><li>PPV &amp; NPV<ul><li>PPV: P(disease|+)</li><li>NPV: P(normal|-)</li></ul></li><li><p>confusion matrix:</p><pre><code>            +                -</code></pre><p>disease   true positive     false negative    → #(+ and disease)/#disease = sensitvity<br>normal    false positive    true negative     → #(- and normal)/#normal = specificity</p><pre><code>           ↓                  ↓</code></pre><p>#(+ and disease)/#(+) = PPV     #(- and normal)/#(-) = NPV</p><p>sensitivity = tp/(tp+fn)<br>specificity = tn/(fp+tn)<br>ppv = tp/(tp+fp)<br>npv = tn/(fn+tn)</p></li><li><p>probability</p><ul><li>ppv = p(pos|pos_p) = p(pos_p|pos)×p(pos)/p(pos_p)<ul><li>sensitivity = p(pos_p|pos)</li><li>prevalence = p(pos)<ul><li>p(pos_p) = truePos+falsePos <ul><li>truePos = p(pos_p|pos)×p(pos) = sensitivity×prevalence</li><li>falsePos = p(pos_p|neg)×p(neg)<ul><li>p(pos_p|neg) = 1-specificity</li><li>p(neg) = 1-prevalence</li></ul></li></ul></li></ul></li></ul></li><li>ppv = sentsitivity×prevalence/(sensitivity×prevalence+(1-specificity)×(1-prevalence))</li></ul></li></ol><h1 id="AI-for-Medical-Prognosis"><a href="#AI-for-Medical-Prognosis" class="headerlink" title="AI for Medical Prognosis"></a>AI for Medical Prognosis</h1><ol><li><p>Linear model</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"># Import the module &apos;LinearRegression&apos; from sklearn</span><br><span class="line">from sklearn.linear_model import LinearRegression</span><br><span class="line"></span><br><span class="line"># Create an object of type LinearRegression</span><br><span class="line">model = LinearRegression()</span><br><span class="line"></span><br><span class="line"># Import the load_data function from the utils module</span><br><span class="line">from utils import load_data</span><br><span class="line"></span><br><span class="line"># Generate features and labels using the imported function</span><br><span class="line">X, y = load_data(100)</span><br><span class="line"></span><br><span class="line">model.fit(X, y)</span><br></pre></td></tr></table></figure><ul><li>Logistic Regression: <code>sklearn.linear_model.LogisticRegression</code></li></ul></li><li><p>Risk score</p><ul><li>Atrial fibrillation: Chads-vasc score</li><li>Liver disease: MELD score</li><li>Heart disease: ASCVD score</li></ul></li><li><p>Measurements</p><ul><li><script type="math/tex; mode=display">c-index = (conordant-0.5\*ties)/permission</script></li></ul></li><li><p>Decision Tree</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line">import shap</span><br><span class="line">import sklearn</span><br><span class="line">import itertools</span><br><span class="line">import pydotplus</span><br><span class="line">import numpy as np</span><br><span class="line">import pandas as pd</span><br><span class="line">import seaborn as sns</span><br><span class="line">import matplotlib.pyplot as plt</span><br><span class="line">from IPython.display import Image </span><br><span class="line">from sklearn.tree import export_graphviz</span><br><span class="line">from sklearn.externals.six import StringIO</span><br><span class="line">from sklearn.tree import DecisionTreeClassifier</span><br><span class="line">from sklearn.ensemble import RandomForestClassifier</span><br><span class="line">from sklearn.model_selection import train_test_split</span><br><span class="line">from sklearn.experimental import enable_iterative_imputer</span><br><span class="line">from sklearn.impute import IterativeImputer, SimpleImputer</span><br><span class="line">#</span><br><span class="line"># We&apos;ll also import some helper functions that will be useful later on.</span><br><span class="line">from util import load_data, cindex</span><br><span class="line">  from sklearn.tree import DecisionTreeClassifier</span><br><span class="line">  dt = DecisionTreeClassifier()</span><br><span class="line">  dt.fit(X,y)</span><br><span class="line">  dt = DecisionTreeClassifier(criterion=&apos;entropy&apos;,</span><br><span class="line">                            max_depth=10,</span><br><span class="line">                            min_samples_split=2</span><br><span class="line">                           )</span><br></pre></td></tr></table></figure><ul><li><p>控制树的深度以处理overfitting</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">dt_hyperparams = &#123;</span><br><span class="line">    &apos;max_depth&apos;:3,</span><br><span class="line">    &apos;min_samples_split&apos;:2</span><br><span class="line">&#125;</span><br><span class="line">dt_reg = DecisionTreeClassifier(**dt_hyperparams, random_state=10)</span><br><span class="line">dt_reg.fit(X_train_dropped, y_train_dropped)</span><br><span class="line">y_train_preds = dt_reg.predict_proba(X_train_dropped)[:, 1]</span><br><span class="line">y_val_preds = dt_reg.predict_proba(X_val_dropped)[:, 1]</span><br><span class="line">print(f&quot;Train C-Index: &#123;cindex(y_train_dropped.values, y_train_preds)&#125;&quot;)</span><br><span class="line">print(f&quot;Val C-Index (expected &gt; 0.6): &#123;cindex(y_val_dropped.values, y_val_preds)&#125;&quot;)</span><br><span class="line">#</span><br><span class="line"># print tree</span><br><span class="line">from sklearn.externals.six import StringIO</span><br><span class="line">dot_data = StringIO()</span><br><span class="line">export_graphviz(dt_reg, feature_names=X_train_dropped.columns, out_file=dot_data,  </span><br><span class="line">                filled=True, rounded=True, proportion=True, special_characters=True,</span><br><span class="line">                impurity=False, class_names=[&apos;neg&apos;, &apos;pos&apos;], precision=2)</span><br><span class="line">graph = pydotplus.graph_from_dot_data(dot_data.getvalue())  </span><br><span class="line">Image(graph.create_png())</span><br></pre></td></tr></table></figure></li><li><p>Random forests</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line">rf = RandomForestClassifier(n_estimators=100, random_state=10)</span><br><span class="line">rf.fit(X_train_dropped, y_train_dropped)</span><br><span class="line">#</span><br><span class="line">def random_forest_grid_search(X_train_dropped, y_train_dropped, X_val_dropped, y_val_dropped):</span><br><span class="line">  #</span><br><span class="line">    # Define ranges for the chosen random forest hyperparameters </span><br><span class="line">    hyperparams = &#123;</span><br><span class="line">        &apos;n_estimators&apos;: [1,2,3],</span><br><span class="line">        &apos;max_depth&apos;: [4,5,6],</span><br><span class="line">        &apos;min_samples_leaf&apos;: [1,2,3],</span><br><span class="line">    &#125;    </span><br><span class="line">    fixed_hyperparams = &#123;</span><br><span class="line">        &apos;random_state&apos;: 10,</span><br><span class="line">    &#125;    </span><br><span class="line">    rf = RandomForestClassifier</span><br><span class="line">    best_rf, best_hyperparams = holdout_grid_search(rf, X_train_dropped, y_train_dropped,</span><br><span class="line">                                                    X_val_dropped, y_val_dropped, hyperparams,</span><br><span class="line">                                                    fixed_hyperparams)</span><br><span class="line">    print(f&quot;Best hyperparameters:\n&#123;best_hyperparams&#125;&quot;)</span><br><span class="line">    #</span><br><span class="line">    y_train_best = best_rf.predict_proba(X_train_dropped)[:, 1]</span><br><span class="line">    print(f&quot;Train C-Index: &#123;cindex(y_train_dropped, y_train_best)&#125;&quot;)</span><br><span class="line">    #</span><br><span class="line">    y_val_best = best_rf.predict_proba(X_val_dropped)[:, 1]</span><br><span class="line">    print(f&quot;Val C-Index: &#123;cindex(y_val_dropped, y_val_best)&#125;&quot;)</span><br><span class="line">    #</span><br><span class="line">    # add fixed hyperparamters to best combination of variable hyperparameters</span><br><span class="line">    best_hyperparams.update(fixed_hyperparams)</span><br><span class="line">    #</span><br><span class="line">    return best_rf, best_hyperparams</span><br><span class="line">#</span><br><span class="line">best_rf, best_hyperparams = random_forest_grid_search(X_train_dropped, y_train_dropped, X_val_dropped, y_val_dropped)</span><br></pre></td></tr></table></figure></li></ul></li><li><p>Ensembling </p><ul><li>desision tree </li><li>gradient boosting</li><li>xgboost</li><li>lightGBM</li></ul></li><li><p>NAN</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">df_booleans = df.isnull()</span><br><span class="line">df_booleans.any()</span><br></pre></td></tr></table></figure></li><li><p>Imputation</p><ul><li><p>mean</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">from sklearn.impute import SimpleImputer</span><br><span class="line">mean_imputer = SimpleImputer(missing_values=np.NaN, strategy=&apos;mean&apos;)</span><br><span class="line">mean_imputer.fit(df)</span><br></pre></td></tr></table></figure></li><li><p>regression：age 和 BP 具有线性关系</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">from sklearn.experimental import enable_iterative_imputer</span><br><span class="line">from sklearn.impute import IterativeImputer</span><br><span class="line">reg_imputer = IterativeImputer()</span><br><span class="line">reg_imputer.fit(df)</span><br><span class="line">nparray_imputed_reg = reg_imputer.transform(df)</span><br></pre></td></tr></table></figure></li></ul></li><li><p>SHAP</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">explainer = shap.TreeExplainer(rf_imputed)</span><br><span class="line">i = 0</span><br><span class="line">shap_value = explainer.shap_values(X_test.loc[X_test_risk.index[i], :])[1]</span><br><span class="line">shap.force_plot(explainer.expected_value[1], shap_value, feature_names=X_test.columns, matplotlib=True)</span><br></pre></td></tr></table></figure></li></ol>]]></content>
      
      
      <categories>
          
          <category> AI for Medicine </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
            <tag> Medicine </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Machine Learning 面试题</title>
      <link href="/2020/04/01/ml/"/>
      <url>/2020/04/01/ml/</url>
      
        <content type="html"><![CDATA[<p>关于机器学习笔试面试的题：</p><ol><li><p>判别式模型 &amp; 生成式模型</p><ul><li>判别式模型（Discriminative Model）是直接对条件概率p(y|x;θ)建模。如：线性回归模型、线性判别分析、支持向量机SVM、神经网络等</li><li>生成式模型（Generative Model）则会对x和y的联合分布p(x,y)建模，然后通过贝叶斯公式来求得p(yi|x)，然后选取使得p(yi|x)最大的yi。如：隐马尔可夫模型HMM、朴素贝叶斯模型、高斯混合模型GMM、狄利克雷分布模型（Latent Dirichlet Allocation，LDA）等。</li></ul></li><li><p>激活函数<br> sigmoid 函数映射之后取值范围为（0，1）<br> tanh函数映射之后取值范围（-1，1）<br> Relu函数映射之后取值范围（0，..）大于等于0</p></li><li><p>A,B,C 三个矩阵尺寸分别为 m∗n，n∗p，p∗q，且 <script type="math/tex">m < n < p < q</script>, 计算效率最高的是 (AB)C</p></li><li><p>在统计模式识分类问题中，当先验概率未知时，可以使用 N-P判决和最小最大损失。<br> 在贝叶斯决策中，对于先验概率p(y)，分为已知和未知两种情况。</p><ul><li>p(y)已知，直接使用贝叶斯公式求后验概率即可；</li><li>p(y)未知，可以使用聂曼-皮尔逊决策(N-P决策)来计算决策面。<br>而最大最小损失规则主要就是使用解决最小损失规则时先验概率未知或难以计算的问题的。</li></ul></li><li><p>路径规划</p><ul><li>A*（A-Star)算法是一种静态路网中求解最短路最有效的方法。 公式表示为： f(n)=g(n)+h(n), 其中 f(n) 是从初始点经由节点n到目标点的估价函数， g(n) 是在状态空间中从初始节点到n节点的实际代价， h(n) 是从n到目标节点最佳路径的估计代价。 保证找到最短路径（最优解的）条件，关键在于估价函数h(n)的选取： 估价值 h(n)\&lt;= n 到目标节点的距离实际值，这种情况下，搜索的点数多，搜索范围大，效率低。但能得到最优解。 如果 估价值&gt;实际值,搜索的点数少，搜索范围小，效率高，但不能保证得到最优解。 </li><li>Dijkstra算法 戴克斯特拉算法（英语：Dijkstra’s algorithm，又译迪杰斯特拉算法）由荷兰计算机科学家艾兹赫尔·戴克斯特拉在1956年提出。戴克斯特拉算法使用了广度优先搜索解决赋权有向图的单源最短路径问题。该算法存在很多变体；戴克斯特拉的原始版本找到两个顶点之间的最短路径，但是更常见的变体固定了一个顶点作为源节点然后找到该顶点到图中所有其它节点的最短路径，产生一个最短路径树。 </li></ul></li><li><p>K-Means算法事先确定常数K，常数K意味着最终的聚类类别数，首先随机选定初始点为质心，并通过计算每一个样本与质心之间的相似度(这里为欧式距离)，将样本点归到最相似的类中，接着，重新计算每个类的质心(即为类中心)，重复这样的过程，知道质心不再改变，最终就确定了每个样本所属的类别以及每个类的质心。由于每次都要计算所有的样本与每一个质心之间的相似度，故在大规模的数据集上，K-Means算法的收敛速度比较慢。</p></li><li><p>特征提取算法分为特征选择和特征抽取两大类</p><ul><li><p>特征选择。常采用特征选择方法。常见的六种特征选择方法：</p><ul><li><p>DF(Document Frequency) 文档频率<br>DF:统计特征词出现的文档数量，用来衡量某个特征词的重要性</p></li><li><p>MI(Mutual Information) 互信息法<br>互信息法用于衡量特征词与文档类别直接的信息量。<br>如果某个特征词的频率很低，那么互信息得分就会很大，因此互信息法倾向”低频”的特征词。<br>相对的词频很高的词，得分就会变低，如果这词携带了很高的信息量，互信息法就会变得低效。</p></li><li><p>(Information Gain) 信息增益法<br>通过某个特征词的缺失与存在的两种情况下，语料中前后信息的增加，衡量某个特征词的重要性。</p></li><li><p>CHI(Chi-square) 卡方检验法<br>利用了统计学中的”假设检验”的基本思想：首先假设特征词与类别直接是不相关的<br>如果利用CHI分布计算出的检验值偏离阈值越大，那么更有信心否定原假设，接受原假设的备则假设：特征词与类别有着很高的关联度。<br>卡方检验（χ2 test），是一种常用的特征选择方法，尤其是在生物和金融领域。χ2 用来描述两个事件的独立性或者说描述实际观察值与期望值的偏离程度。χ2值越大，则表明实际观察值与期望值偏离越大，也说明两个事件的相互独立性越弱。 </p></li><li><p>WLLR(Weighted Log Likelihood Ration)加权对数似然</p></li><li><p>WFO（Weighted Frequency and Odds）加权频率和可能性</p></li><li><p>期望交叉熵，以文本分类为例子，期望交叉熵用来度量一个词对于整体的重要程度</p></li></ul></li><li><p>特征抽取（降维）<br>  PCA, SVD等</p></li><li><p>在ID3决策树中，也使用信息增益作为特征选择的方法，在C4.5决策树中，使用信息增益比作为特征选择的方法，在CART中，使用基尼指数作为特征选择的方法    </p></li></ul></li><li><p>Naive Bayes是一种特殊的Bayes分类器,特征变量是X,类别标签是C,它的一个假定是特征变量X的各个维度是类别条件独立随机变量。</p></li><li><p>tp = true positive, tn = true negative, fp = false positive, fn = false negative</p><ul><li>Precision= tp / (tp + fp)</li><li>Recall = tp / (tp + fn)</li></ul></li><li><p>解决隐马模型中预测问题的算法是维特比算法。</p><ul><li>前向、后向算法解决的是一个评估问题，即给定一个模型，求某特定观测序列的概率，用于评估该序列最匹配的模型。</li><li>Baum-Welch算法解决的是一个模型训练问题，即参数估计，是一种无监督的训练方法，主要通过EM迭代实现；</li><li>维特比算法解决的是给定 一个模型和某个特定的输出序列，求最可能产生这个输出的状态序列。如通过海藻变化（输出序列）来观测天气（状态序列），是预测问题，通信中的解码问题。</li></ul></li><li><p>N-gram是一种简单有效的统计语言模型，通常n采用1-3之间的值，它们分别称为unigram、bigram和trigram。现有给定训练语料合计三个文档如下：<br>D1： John read Moby Dick<br>D2： Mary read a different book,<br>D3： She read a book by Cher<br>利用bigram求出句子“John read a book”的概率大约是0.06</p><p>2-gram公式</p><script type="math/tex; mode=display">P(s1,s2,s3...) = P(s1)*P(s2|s1)*P(s3|s2).....</script><p>解：<br>john在文章开头的概率：P（john） = 1/3<br>P（read | John） = 1<br>P(a|read) = 2/3<br>P(book|a) = 1/2<br>P(尾巴|book) = 1/2, book出现两次，其中一次是在句子结尾处<br>P(“John read a book”) = <script type="math/tex">1/3*1*2/3*1/2*1/2 = 1/18 ≈ 0.06</script></p></li><li><p>如果假设h在n=65的独立抽取样本上出现r=10个错误，真实的错误率的90%的置信区间（双侧的）是0.16±0.073。</p><script type="math/tex; mode=display">error_D(h)$$ 的 N% 的置信区间：$$ errors(h) \pm zv \sqrt{\frac{errors(h)(1-erros(h))}{n}}</script></li><li><p>正则化：</p><ul><li>正则化可以防止过拟合</li><li>L1正则化能得到稀疏解</li><li>L2正则化约束了解空间</li><li>Dropout也是一种正则化方法</li></ul></li><li><p>可以用来完成命名实体的任务的算法模型：HMM, CRF, LSTM, seq2seq</p><ul><li>基于规则的方法。根据语言学上预定义的规则。但是由于语言结构本身的不确定性，规则的制定上难度较大。</li><li>基于统计学的方法。利用统计学找出文本中存在的规律。主要有隐马尔可夫(HMM)、条件随机场(CRF)模型和Viterbi算法、支持向量机（Support Vector Machine, SVM）。</li><li>神经网络。 LSTM+CRF模型，基于RNN的seq2seq模型</li><li>LDA是主题模型，概括文本摘要啥的</li></ul></li><li><p>激活函数是非线性的，y=2x不可以做激活函数。</p></li><li><p>朴素贝叶斯定理体现了后验概率 P(y|x) 、先验概率 P(y) 、条件概率 P(x|y)之间的关系：<script type="math/tex">P(y|x)=P(x|y)·P(y)/P(x)</script><br>朴素贝叶斯之所以叫“朴素”是因为它假设输入的不同特征之间是独立的。构建朴素贝叶斯分类器的步骤如下：</p><ul><li>根据训练样例分别计算每个类别出现的概率P(yi)， </li><li>对每个特征属性计算所有划分的条件概率P(xi|yi), </li><li>对每个类别计算 <script type="math/tex">P(X|yi)\*P(yi)</script>, </li><li>选择3步骤中数值最大项作为X的类别yk。</li></ul></li><li><p>resnet-34 和 resnet-50 都是有四组block，每组分别是 3 4 6 3个block，resnet-34 的每个block里面有两个卷积层，resnet-50的每个block里面有三个，另外这两个网络的最开始都有一个单独的卷积层<br>(3+4+6+3)*3+1=49</p></li><li><p>数据挖掘的挖掘方法包括:决策树 、神经网络 、回归 、聚类 、关联规则 、贝叶斯分类</p></li><li><p>类别不平衡（class-imbanlance）就是指分类问题中不同类别的训练样本相差悬殊的情况，例如正例有900个，而反例只有100个，这个时候我们就需要进行相应的处理来平衡这个问题：</p><ul><li>在训练样本较多的类别中进行欠采样</li><li>直接基于原数据集进行学习，对预测值进行再缩放处理（weighted loss)</li><li>通过对反例中的数据进行插值，来产生额外的反例</li></ul></li><li><p>置信度计算规则为： 同时购买商品A和商品B的交易次数/购买了商品A的次数<br>支持度计算规则为： 同时购买了商品A和商品B的交易次数/总的交易次数<br>支持度表示 Support(X→Y) = P(X,Y) / P(I) = P(X∩Y) / P(I) = num(X∩Y) / num(I)<br>可信度表示 Confidence(X→Y) = P(Y|X)  = P(X,Y) / P(X) = P(X∩Y) / P(X)<br>提升度表示 可信度/支持度</p></li><li><p>Adaboost算法的思想是在前一轮识别过程中识别错误的样本会在下一轮中提升权重，而那些识别正确的样本会降低权重。所以不是独立的学习弱分类器。<br>AdaBoost模型是弱分类器的线性组合。<br>提升树是以分类树或者回归树为基本分类器的提升办法，提升树被认为是统计学习中最有效的办法之一。<br>AdaBoost算法的一个解释是该算法实际上是前向分步算法的一个实现，在这个方法里，模型是加法模型，损失函数是指数损失，算法是前向分步算法。</p></li><li><p>假如使用一个较复杂的脊回归模型 (Ridge Regression)，来拟合样本数据时，通过调整正则化参数λ，来调整模型复杂度。当λ较大时，当λ增大时，偏差增大，方差减小。<br>.λ越大，对模型中参数的惩罚力度越大，因此会有更多的参数被训练为0，模型也就变得更加简单了。模型复杂度越低，方差小，但偏差大。</p></li><li><p>CNN常见的Loss函数:softmax_loss, sigmoid_loss, siamese_loss<br>在传统的siamese network中一般使用Contrastive Loss作为损失函数，这种损失函数可以有效的处理孪生神经网络中的paired data的关系。<a href="https://www.jianshu.com/p/21be99fb58ca" target="_blank" rel="noopener">参考</a></p></li><li><p>优化方法：</p><ul><li>批量梯度下降法在每次参数更新时同时迭代所有样本，优点是迭代次数少，并行计算，缺点是在样本规模大时训练缓慢。</li><li>随机梯度下降法在每次参数更新时迭代一个样本，优点时在样本规模大时训练快，缺点是迭代次数多，且容易收敛到局部最优解；</li><li>牛顿法是一种计算二阶梯度的算法，与梯度下降法相比，收敛速度更快，但计算复杂，每次参数更新都要计算Hession矩阵的逆；</li><li>梯度下降法是利用当前位置的负梯度作为搜索方向的方法。</li><li>牛顿法和梯度下降法相比，一个劣势是求解复杂，一个优势是收敛速度加快。</li><li>共轭梯度法仅需利用一阶导数的信息，但是收敛速度高于梯度下降法</li></ul></li><li><p>AR模型：自回归模型，是一种线性模型<br>MA模型：移动平均法模型，其中使用趋势移动平均法建立直线趋势的预测模型<br>ARMA模型：自回归滑动平均模型，拟合较高阶模型<br>GARCH模型：广义回归模型，对误差的方差建模，适用于波动性的分析和预测</p></li><li><p>spss中交叉分析主要用来检验两个变量之间是否存在关系，或者说是否独立，其零假设为两个变量之间没有关系。在实际工作中，经常用交叉表来分析比例是否相等。例如分析不同的性别对不同的报纸的选择有什么不同。</p></li><li><p>均值移动（Mean Shift）算法的核心思想是：找到概率密度梯度为零的采样点，并以此作为特征空间聚类的模式点。<br>ttps://blog.csdn.net/hjimce/article/details/45718593<br>基于核密度估计的爬山算法，基本形式是：<br>给d维空间的n个数据点集X，那么对于空间中任一点x的meanshift就可以表示为漂移向量（表示圆心的向量），用sk表示数据集的点到x的距离小于球半径h的数据点集。漂移过程就是不断加入新的点（就是到x距离小于求半径的点），一边更新球圆心x。圆心会向数据集密度最大的地方移动。</p></li><li><p>随机抽样一致算法（random sample consensus,RANSAC）,采用迭代的方式从一组包含离群的被观测数据中估算出数学模型的参数。<br>给定一组（通常很小）的内群，存在一个程序，这个程序可以估算最佳解释或最适用于这一数据模型的参数。<br>“内群”数据可以通过几组模型的参数来叙述其分别，而“离群”数据则是不适合模型化的数据。<br>数据会受噪声影响，噪声指的是离群，例如从极端的噪声或错误解释有关数据的测量或不正确的假设</p></li><li><p>BatchNorm 层对于 input batch 会统计出 mean 和 variance 用于计算 EMA。如果input batch 的 shape 为(B, C, H, W)，统计出的 mean 和 variance 的 shape 为: 1*C*1*1<br>BN 层的目的本就是用于归一化，所以显而易见要保证通道数C不变，其余长宽为0</p></li><li><p>p(w1 | xi) = p(xiw1) / p(xi) = p(xi | w1) <em> p(w1) / p(xi)<br>根据贝叶斯定理：P(Y = 1|X = 1) = P(X = 1|Y =1) </em> P(Y = 1)/P(X = 1)<br>根据全概率公式：P(X =1) = P(X = 1|Y = 1) <em> P(Y = 1) + P(X = 1|Y = 0) </em> P(Y = 0) </p></li><li><p>基于核的机器学习算法: Radial Basis Function, Linear Discrimimate Analysis, Support Vector Machine</p></li><li><p>SPSS的界面中，主窗口是数据编辑窗口<br>SPSS中，数据整理的功能主要集中在数据和转换等菜单中</p></li><li><p>假设你需要调整超参数来最小化代价函数（cost function），可使用：穷举搜索，随机搜索，Bayesian优化</p></li><li><p>假设你有5个大小为7x7、边界值为0的卷积核，同时卷积神经网络第一层的深度为1。此时如果你向这一层传入一个维度为224x224x3的数据，那么神经网络下一层所接收到的数据维度是218x218x5<br>(input - kernel_width+1) / step_length</p></li><li><p>关于支持向量机SVM：<br>L2正则项，作用是最大化分类间隔，使得分类器拥有更强的泛化能力<br>Hinge 损失函数，作用是最小化经验分类错误<br>当参数C越小时，分类间隔越大，分类错误越多，趋于欠学习<br>间隔应该是2/||w||才对，向量的模通常指的就是||w||。</p></li><li><p>随机森林不需要剪枝，因为本来就很大方差，防止过拟合<br>GBDT核心在于每一棵树学的是之前所有树的结论和的残差，残差是一个加预测值后能得到真实值得累加量。<br>xgboost和GBDT差不多，不过还支持线性分类器<br>xgboost可以自定损失函数，速度很快，但是对异常值很敏感。<br>SVM的目标是找到使得训练数据尽可能分开且分类间隔最大的超平面，属于结构风险最小化<br>Naive Bayes是一种特殊的Bayes分类器，其一个假定是每个变量相互独立。</p></li><li><p>输入图片大小为200×200，依次经过一层卷积（kernel size 5×5，padding 1，stride 2），pooling（kernel size 3×3，padding 0，stride 1），又一层卷积（kernel size 3×3，padding 1，stride 1）之后，输出特征图大小为97。<br>计算尺寸不被整除只在GoogLeNet中遇到过。卷积向下取整，池化向上取整。<br>本题 （200-5+2*1）/2+1 为99.5，取99<br>（99-3）/1+1 为97<br>（97-3+2*1）/1+1 为97<br>研究过网络的话看到stride为1的时候，当kernel为 3 padding为1或者kernel为5 padding为2 一看就是卷积前后尺寸不变。<br>计算GoogLeNet全过程的尺寸也一样。<br>输出尺寸=(输入尺寸-filter尺寸+2*padding）/stride+1</p></li><li><p>在机器学习中需要划分数据集，常用的划分测试集和训练集的划分方法有:留出法,交叉验证法,自助法<br>自助法，又称为自助抽样法</p></li><li><p>决策树常用三种指标来确定是否继续划分集合：信息增益、信息增益率，基尼指数。<br>信息熵：即数据样本的纯度，纯度越高，熵越小。<br>信息增益：按照某一特征划分数据集后熵的减少量，选择减少量最多的特征进行划分，但是偏好特征取值较多的特征，常见模型ID3。<br>信息增益率：在信息怎亿的基础上除以一个固有值（intrinsic value，和取值数目有关），会对取值数目较多的特征有更多惩罚，偏好取值数较少的特征，常见模型C4.5<br>基尼指数：从样本集D中随机抽取两个样本，其类别标记不一致的概率，因此越小越好<br>ID3决策树是根据信息增益来划分属性<br>C4.5决策树是根据增益率来划分属性<br>CART决策树是根据基尼指数来划分属性</p></li><li><p>分支定界法类似决策树的决策特征，要选择那些具有强可分辨性的少量特征。<br>在选用的可分性判据J对特征数目单调不减情况下,用分支定界法做特征选择计算量相对较少<br>C_n^d &gt;&gt; n, n为原特征个数，d为要选出来的特征个数</p></li><li><p>激活函数的特性：</p><ol><li>非线性：<br>即导数不是常数。这个条件是多层神经网络的基础，保证多层网络不退化成单层线性网络。这也是激活函数的意义所在。</li><li>几乎处处可微：<br>可微性保证了在优化中梯度的可计算性。传统的激活函数如sigmoid等满足处处可微。对于分段线性函数比如ReLU，只满足几乎处处可微（即仅在有限个点处不可微）。对于SGD算法来说，由于几乎不可能收敛到梯度接近零的位置，有限的不可微点对于优化结果不会有很大影响。</li><li>计算简单：<br>非线性函数有很多。极端的说，一个多层神经网络也可以作为一个非线性函数，类似于Network In Network中把它当做卷积操作的做法。但激活函数在神经网络前向的计算次数与神经元的个数成正比，因此简单的非线性函数自然更适合用作激活函数。这也是ReLU之流比其它使用Exp等操作的激活函数更受欢迎的其中一个原因。</li><li>非饱和性（saturation）：<br>饱和指的是在某些区间梯度接近于零（即梯度消失），使得参数无法继续更新的问题。最经典的例子是Sigmoid，它的导数在x为比较大的正值和比较小的负值时都会接近于0。更极端的例子是阶跃函数，由于它在几乎所有位置的梯度都为0，因此处处饱和，无法作为激活函数。ReLU在x&gt;0时导数恒为1，因此对于再大的正值也不会饱和。但同时对于x&lt;0，其梯度恒为0，这时候它也会出现饱和的现象（在这种情况下通常称为dying ReLU）。Leaky ReLU和PReLU的提出正是为了解决这一问题。</li><li>单调性（monotonic）：<br>即导数符号不变。这个性质大部分激活函数都有，除了诸如sin、cos等。个人理解，单调性使得在激活函数处的梯度方向不会经常改变，从而让训练更容易收敛。</li><li>输出范围有限：<br>有限的输出范围使得网络对于一些比较大的输入也会比较稳定，这也是为什么早期的激活函数都以此类函数为主，如Sigmoid、TanH。但这导致了前面提到的梯度消失问题，而且强行让每一层的输出限制到固定范围会限制其表达能力。因此现在这类函数仅用于某些需要特定输出范围的场合，比如概率输出（此时loss函数中的log操作能够抵消其梯度消失的影响）、LSTM里的gate函数。</li><li>接近恒等变换（identity）：<br>即约等于x。这样的好处是使得输出的幅值不会随着深度的增加而发生显著的增加，从而使网络更为稳定，同时梯度也能够更容易地回传。这个与非线性是有点矛盾的，因此激活函数基本只是部分满足这个条件，比如TanH只在原点附近有线性区（在原点为0且在原点的导数为1），而ReLU只在x&gt;0时为线性。这个性质也让初始化参数范围的推导更为简单。这种恒等变换的性质也被其他一些网络结构设计所借鉴，比如CNN中的ResNet和RNN中的LSTM。</li><li>参数少：<br>大部分激活函数都是没有参数的。像PReLU带单个参数会略微增加网络的大小。还有一个例外是Maxout，尽管本身没有参数，但在同样输出通道数下k路Maxout需要的输入通道数是其它函数的k倍，这意味着神经元数目也需要变为k倍；但如果不考虑维持输出通道数的情况下，该激活函数又能将参数个数减少为原来的k倍。<br>归一化（normalization）：<br>这个是最近才出来的概念，对应的激活函数是SELU，主要思想是使样本分布自动归一化到零均值、单位方差的分布，从而稳定训练。在这之前，这种归一化的思想也被用于网络结构的设计，比如Batch Normalization。</li></ol></li><li><p>HK算法思想很朴实,就是在最小均方误差准则下求得权矢量.<br>他相对于感知器算法的优点在于,他适用于线性可分和非线性可分得情况,对于线性可分的情况,给出最优权矢量,对于非线性可分得情况,能够判别出来,以退出迭代过程.</p></li><li><p><a href="https://zhuanlan.zhihu.com/p/32626442" target="_blank" rel="noopener">深度学习优化算法</a></p><ul><li>Gradient Desent: 梯度下降<ul><li>SGD: 收敛慢，可能在鞍点处震荡，在如何选择学习率上有难点。</li></ul></li><li>Momentum: 在SGD上引入动量，加速 SGD 在正确方向的下降并抑制震荡。 <ul><li>SGD-M： 在原步长之上，增加了与上一时刻步长相关的优化内容。</li></ul></li><li>Nesterov Accelerated Gradient (NAG) :在目标函数有增高趋势之前，减缓更新速率。</li><li>Adagrad: 引入了二阶动量。其二阶动量的对应分量较大，学习率就较小。这一方法在稀疏数据的场景下表现很好。</li><li>RMSprop: 只关注最近某一时间窗口内的下降梯度。</li><li><a href="https://www.jiqizhixin.com/graph/technologies/173c1ba6-0a13-45f6-9374-ec0389124832" target="_blank" rel="noopener">Adadelta</a>: 不积累所有过去的平方梯度，而是将积累的过去梯度的窗口限制在一定的大小。</li><li>Adam: RMSprop 和 Momentum 的结合。和 RMSprop 对二阶动量使用指数移动平均类似，Adam 中对一阶动量也是用指数移动平均计算。</li><li>NAdam: 在 Adam 之上融合了 NAG 的思想。</li><li><a href="https://keras.io/zh/optimizers/" target="_blank" rel="noopener">keras实现</a></li></ul></li><li><p><a href="https://zhuanlan.zhihu.com/p/31470216" target="_blank" rel="noopener">LDA主题模型</a></p></li><li><p>RNN, LSTM, GRU</p><ul><li><a href="https://zhuanlan.zhihu.com/p/48760897" target="_blank" rel="noopener">LSTM vs. GRU</a>: 遗忘门、更新门和输出门</li><li><a href="https://zhuanlan.zhihu.com/p/32481747" target="_blank" rel="noopener">GRU</a>: 重置门和更新门</li></ul></li><li><p><a href="https://zh.wikipedia.org/wiki/%E9%A9%AC%E5%B0%94%E5%8F%AF%E5%A4%AB%E9%93%BE" target="_blank" rel="noopener">马尔可夫链</a><br>马尔可夫链是满足马尔可夫性质的随机变量序列X1, X2, X3, …，即给出当前状态，将来状态和过去状态是相互独立的。</p></li><li><p><a href="https://zh.wikipedia.org/wiki/%E5%8F%97%E9%99%90%E7%8E%BB%E5%B0%94%E5%85%B9%E6%9B%BC%E6%9C%BA" target="_blank" rel="noopener">受限玻尔兹曼机</a></p></li><li><p><a href="https://zh.wikipedia.org/wiki/%E5%85%B3%E8%81%94%E8%A7%84%E5%88%99%E5%AD%A6%E4%B9%A0" target="_blank" rel="noopener">关联规则学习</a><br>假设 <script type="math/tex">I=\{I_{1},I_{2},\ldots ,I_{m}\}</script> 是项的集合。给定一个交易数据库 <script type="math/tex">D=\{t_{1},t_{2},\ldots ,t_{n}\}</script>，其中每个事务（Transaction）t是I的非空子集，即<script type="math/tex">t\subseteq I</script>，每一个交易都与一个唯一的标识符TID（Transaction ID）对应。关联规则是形如<script type="math/tex">X\Rightarrow Y</script>的蕴涵式，其中<script type="math/tex">X,Y\subseteq I</script> 且 <script type="math/tex">X\cap Y=\emptyset</script> ，<script type="math/tex">X</script> 和 <script type="math/tex">Y</script> 分别称为关联规则的先导(antecedent或left-hand-side, LHS)和后继(consequent或right-hand-side, RHS) 。关联规则 <script type="math/tex">X\Rightarrow Y</script> 在 <script type="math/tex">D</script>中的支持度（support）是D中事务包含 <script type="math/tex">X\cup Y</script> 的百分比，即概率 <script type="math/tex">P(X\cup Y)</script>；置信度（confidence）是包含X的事务中同时包含Y的百分比，即条件概率<script type="math/tex">P\left(Y|X\right)</script>。如果同时满足最小支持度阈值和最小置信度阈值，则认为关联规则是有趣的。这些阈值由用户或者专家设定。</p></li><li><p><a href="https://keras.io/applications/" target="_blank" rel="noopener">Keras applications</a><br>提供了如 VGG, RESNET, <a href="https://blog.csdn.net/qq_38807688/article/details/84589563" target="_blank" rel="noopener">Inceptionv3</a>等网络模型。</p></li><li><p><a href="https://keras.io/zh/losses/" target="_blank" rel="noopener">Keras 损失函数</a></p><ul><li>mean squared error</li><li>mean absolute error</li><li>hinge</li><li>categorical crossentropy</li><li>sparse categorical crossentropy</li><li>binary crossentropy</li></ul></li><li><p><a href="https://keras.io/zh/metrics/" target="_blank" rel="noopener">keras 评估标准</a></p><ul><li>binary accuracy</li><li>categorical accuracy</li><li>sparse categorical accuracy</li><li>top-k categorical accuracy</li><li>sparse top-k categorical accuracy</li></ul></li><li><p>二叉树遍历：</p><ul><li>前序遍历：根结点 —-&gt; 左子树 —-&gt; 右子树</li><li>中序遍历：左子树—-&gt; 根结点 —-&gt; 右子树</li><li>后序遍历：左子树 —-&gt; 右子树 —-&gt; 根结点</li><li>层次遍历：只需按层次遍历即可</li></ul></li><li><p><a href="https://zhuanlan.zhihu.com/p/29247151" target="_blank" rel="noopener">神经网络不work</a>：</p><ul><li>数据规范化：从数据中减去均值再除以标准差</li><li>可视化过程：检查网络是否有效</li><li>数据预处理</li><li>正则化：</li><li>批次大小</li><li>学习率</li><li>激活函数</li><li>不良梯度：relu-&gt;leaky relu or elu</li><li>初始化网络权重</li><li>网络层数</li><li>隐神经元个数</li></ul></li></ol>]]></content>
      
      
      <categories>
          
          <category> Machine Learning </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Tensorflow 2.0</title>
      <link href="/2020/03/30/tensorflow-2-0/"/>
      <url>/2020/03/30/tensorflow-2-0/</url>
      
        <content type="html"><![CDATA[<p>Tensorflow 2.x的学习笔记。主要教程为<a href="https://github.com/dragen1860/Deep-Learning-with-TensorFlow-book">Deep-Learning-with-TensorFlow-book</a>，<a href="https://hecongqing.gitbook.io/tfnotes/" target="_blank" rel="noopener">tensorflow 2.0 实战笔记</a></p><h2 id="Tensorflow-2-x-vs-1-x"><a href="#Tensorflow-2-x-vs-1-x" class="headerlink" title="Tensorflow 2.x vs. 1.x"></a>Tensorflow 2.x vs. 1.x</h2><ul><li><p>Tensorflow 1.x 先创建计算图，然后运行，为静态图模式，该编程方法叫做符号式编程。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">import tensorflow as tf</span><br><span class="line"># 1.创建计算图阶段</span><br><span class="line"># 创建2个输入端子，指定类型和名字</span><br><span class="line">a_ph = tf.placeholder(tf.float32, name=&apos;variable_a&apos;)</span><br><span class="line">b_ph = tf.placeholder(tf.float32, name=&apos;variable_b&apos;)</span><br><span class="line"># 创建输出端子的运算操作，并命名</span><br><span class="line">c_op = tf.add(a_ph, b_ph, name=&apos;variable_c&apos;)</span><br><span class="line"></span><br><span class="line"># 2.运行计算图阶段</span><br><span class="line"># 创建运行环境</span><br><span class="line">sess = tf.InteractiveSession()</span><br><span class="line"># 初始化步骤也需要作为操作运行</span><br><span class="line">init = tf.global_variables_initializer()</span><br><span class="line">sess.run(init) # 运行初始化操作，完成初始化</span><br><span class="line"># 运行输出端子，需要给输入端子赋值</span><br><span class="line">c_numpy = sess.run(c_op, feed_dict=&#123;a_ph: 2., b_ph: 4.&#125;)</span><br><span class="line"># 运算完输出端子才能得到数值类型的c_numpy</span><br><span class="line">print(&apos;a+b=&apos;,c_numpy)</span><br></pre></td></tr></table></figure></li><li><p>Tensorflow 2.x 支持动态图优先模式，即为命令式编程。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">import tensorflow as tf</span><br><span class="line"># 1.创建输入张量</span><br><span class="line">a = tf.constant(2.)</span><br><span class="line">b = tf.constant(4.)</span><br><span class="line"># 2.直接计算并打印</span><br><span class="line">print(&apos;a+b=&apos;,a+b)</span><br></pre></td></tr></table></figure></li></ul><h2 id="核心功能"><a href="#核心功能" class="headerlink" title="核心功能"></a>核心功能</h2><ul><li><p>加速计算（GPU vs. CPU)</p>  <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"># 创建在CPU 上运算的2个矩阵</span><br><span class="line">with tf.device(&apos;/cpu:0&apos;):</span><br><span class="line">cpu_a = tf.random.normal([1, n])</span><br><span class="line">cpu_b = tf.random.normal([n, 1])</span><br><span class="line">print(cpu_a.device, cpu_b.device)</span><br><span class="line"># 创建使用GPU 运算的2 个矩阵</span><br><span class="line">with tf.device(&apos;/gpu:0&apos;):</span><br><span class="line">gpu_a = tf.random.normal([1, n])</span><br><span class="line">gpu_b = tf.random.normal([n, 1])</span><br><span class="line">print(gpu_a.device, gpu_b.device)</span><br><span class="line"></span><br><span class="line">def cpu_run():</span><br><span class="line">with tf.device(&apos;/cpu:0&apos;):</span><br><span class="line">c = tf.matmul(cpu_a, cpu_b)</span><br><span class="line">return c</span><br><span class="line">def gpu_run():</span><br><span class="line">with tf.device(&apos;/gpu:0&apos;):</span><br><span class="line">c = tf.matmul(gpu_a, gpu_b)</span><br><span class="line">return c</span><br><span class="line"></span><br><span class="line"># 第一次计算需要热身，避免将初始化阶段时间结算在内</span><br><span class="line">cpu_time = timeit.timeit(cpu_run, number=10)</span><br><span class="line">gpu_time = timeit.timeit(gpu_run, number=10)</span><br><span class="line">print(&apos;warmup:&apos;, cpu_time, gpu_time)</span><br><span class="line"># 正式计算10 次，取平均时间</span><br><span class="line">cpu_time = timeit.timeit(cpu_run, number=10)</span><br><span class="line">gpu_time = timeit.timeit(gpu_run, number=10)</span><br><span class="line">print(&apos;run time:&apos;, cpu_time, gpu_time)</span><br></pre></td></tr></table></figure><p>  当 <script type="math/tex">n>10^4</script> 后，CPU计算速度明显上升。</p></li><li><p>自动梯度</p>  <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">import tensorflow as tf</span><br><span class="line"># 创建4 个张量</span><br><span class="line">a = tf.constant(1.)</span><br><span class="line">b = tf.constant(2.)</span><br><span class="line">c = tf.constant(3.)</span><br><span class="line">w = tf.constant(4.)</span><br><span class="line">with tf.GradientTape() as tape:# 构建梯度环境</span><br><span class="line">tape.watch([w]) # 将w 加入梯度跟踪列表</span><br><span class="line"># 构建计算过程</span><br><span class="line">y = a * w**2 + b * w + c</span><br><span class="line"># 求导</span><br><span class="line">[dy_dw] = tape.gradient(y, [w])</span><br><span class="line">print(dy_dw) # 打印出导数</span><br><span class="line"># tf.Tensor(10.0, shape=(), dtype=float32)</span><br></pre></td></tr></table></figure></li><li><p>常用神经网络接口</p></li></ul><h2 id="回归问题"><a href="#回归问题" class="headerlink" title="回归问题"></a>回归问题</h2><ul><li>神经元线性模型：<script type="math/tex; mode=display">y=wx+b</script></li><li>平方和误差：<script type="math/tex; mode=display">L = \frac{1}{n} \sum_{i=1}^{n}(wx^{(i)}+b-y^{(i)})^2</script></li><li>最优参数：<script type="math/tex; mode=display">w^*, b^* = \mathop{\arg\min}_{w,b} \frac{1}{n} \sum_{i=1}^{n}(wx^{(i)}+b-y^{(i)})^2</script></li><li>梯度下降法：迭代更新<script type="math/tex; mode=display">w' = w - \eta \frac{\partial L}{\partial w}</script><script type="math/tex; mode=display">b' = w- \eta \frac{\partial L}{\partial b}</script></li><li><p>实例：</p><ul><li><p>采集数据： <script type="math/tex">y=1.477x+0.089+\epsilon, \epsilon \sim \mathbb{N}(0,0.01)</script></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">data = []# 保存样本集的列表</span><br><span class="line">for i in range(100): # 循环采样100 个点</span><br><span class="line">x = np.random.uniform(-10., 10.) # 随机采样输入x</span><br><span class="line"># 采样高斯噪声</span><br><span class="line">eps = np.random.normal(0., 0.1)</span><br><span class="line"># 得到模型的输出</span><br><span class="line">y = 1.477 * x + 0.089 + eps</span><br><span class="line">data.append([x, y]) # 保存样本点</span><br><span class="line">data = np.array(data) # 转换为2D Numpy 数组</span><br></pre></td></tr></table></figure></li><li><p>计算误差</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">def mse(b, w, points):</span><br><span class="line"># 根据当前的w,b 参数计算均方差损失</span><br><span class="line">totalError = 0</span><br><span class="line">for i in range(0, len(points)): # 循环迭代所有点</span><br><span class="line">x = points[i, 0] # 获得i 号点的输入x</span><br><span class="line">y = points[i, 1] # 获得i 号点的输出y</span><br><span class="line"># 计算差的平方，并累加</span><br><span class="line">totalError += (y - (w * x + b)) ** 2</span><br><span class="line"># 将累加的误差求平均，得到均方差</span><br><span class="line">return totalError / float(len(points))</span><br></pre></td></tr></table></figure></li><li><p>计算梯度</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line">def step_gradient(b_current, w_current, points, lr):</span><br><span class="line"># 计算误差函数在所有点上的导数，并更新w,b</span><br><span class="line">b_gradient = 0</span><br><span class="line">w_gradient = 0</span><br><span class="line">M = float(len(points)) # 总样本数</span><br><span class="line">for i in range(0, len(points)):</span><br><span class="line">x = points[i, 0]</span><br><span class="line">y = points[i, 1]</span><br><span class="line"># 误差函数对b 的导数：grad_b = 2(wx+b-y)，参考公式(2.3)</span><br><span class="line">b_gradient += (2/M) * ((w_current * x + b_current) - y)</span><br><span class="line"># 误差函数对w 的导数：grad_w = 2(wx+b-y)*x，参考公式(2.2)</span><br><span class="line">w_gradient += (2/M) * x * ((w_current * x + b_current) - y)</span><br><span class="line"># 根据梯度下降算法更新 w&apos;,b&apos;,其中lr 为学习率</span><br><span class="line">new_b = b_current - (lr * b_gradient)</span><br><span class="line">new_w = w_current - (lr * w_gradient)</span><br><span class="line">return [new_b, new_w]</span><br></pre></td></tr></table></figure></li><li><p>梯度更新</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">def gradient_descent(points, starting_b, starting_w, lr, num_iterations):</span><br><span class="line"># 循环更新w,b 多次</span><br><span class="line">b = starting_b # b 的初始值</span><br><span class="line">w = starting_w # w 的初始值</span><br><span class="line"># 根据梯度下降算法更新多次</span><br><span class="line">for step in range(num_iterations):</span><br><span class="line"># 计算梯度并更新一次</span><br><span class="line">b, w = step_gradient(b, w, np.array(points), lr)</span><br><span class="line">loss = mse(b, w, points) # 计算当前的均方差，用于监控训练进度</span><br><span class="line">if step%50 == 0: # 打印误差和实时的w,b 值</span><br><span class="line">print(f&quot;iteration:&#123;step&#125;, loss:&#123;loss&#125;, w:&#123;w&#125;, b:&#123;b&#125;&quot;)</span><br><span class="line">return [b, w] # 返回最后一次的w,b</span><br></pre></td></tr></table></figure></li><li><p>主函数</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">def main():</span><br><span class="line"># 加载训练集数据，这些数据是通过真实模型添加观测误差采样得到的</span><br><span class="line">lr = 0.01 # 学习率</span><br><span class="line">initial_b = 0 # 初始化b 为0</span><br><span class="line">initial_w = 0 # 初始化w 为0</span><br><span class="line">num_iterations = 1000</span><br><span class="line"># 训练优化1000 次，返回最优w*,b*和训练Loss 的下降过程</span><br><span class="line">[b, w], losses = gradient_descent(data, initial_b, initial_w, lr, num_it</span><br><span class="line">erations)</span><br><span class="line">loss = mse(b, w, data) # 计算最优数值解w,b 上的均方差</span><br><span class="line">print(f&apos;Final loss:&#123;loss&#125;, w:&#123;w&#125;, b:&#123;b&#125;&apos;)</span><br></pre></td></tr></table></figure></li></ul></li></ul><h2 id="分类问题"><a href="#分类问题" class="headerlink" title="分类问题"></a>分类问题</h2><ul><li>非线性模型<br>激活函数： sigmoid, relu<script type="math/tex; mode=display">o = \sigma(wx+b)</script></li><li><p>下载MNIST数据:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">import os</span><br><span class="line">import tensorflow as tf # 导入TF 库</span><br><span class="line">from tensorflow import keras # 导入TF 子库</span><br><span class="line">from tensorflow.keras import layers, optimizers, datasets # 导入TF 子库</span><br><span class="line"></span><br><span class="line">(x, y), (x_val, y_val) = datasets.mnist.load_data() # 加载数据集</span><br><span class="line">x = 2*tf.convert_to_tensor(x, dtype=tf.float32)/255.-1 # 转换为张量，缩放到-1~1</span><br><span class="line">y = tf.convert_to_tensor(y, dtype=tf.int32) # 转换为张量</span><br><span class="line">y = tf.one_hot(y, depth=10) # one-hot 编码</span><br><span class="line">print(x.shape, y.shape)</span><br><span class="line">train_dataset = tf.data.Dataset.from_tensor_slices((x, y)) # 构建数据集对象</span><br><span class="line">train_dataset = train_dataset.batch(512) # 批量训练</span><br></pre></td></tr></table></figure></li><li><p>网络搭建</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">model = keras.Sequential([ # 3 个非线性层的嵌套模型</span><br><span class="line">layers.Dense(256, activation=&apos;relu&apos;),</span><br><span class="line">layers.Dense(128, activation=&apos;relu&apos;),</span><br><span class="line">layers.Dense(10)])</span><br></pre></td></tr></table></figure></li><li><p>模型训练</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">with tf.GradientTape() as tape: # 构建梯度记录环境</span><br><span class="line"># 打平，[b, 28, 28] =&gt; [b, 784]</span><br><span class="line">x = tf.reshape(x, (-1, 28*28))</span><br><span class="line"># Step1. 得到模型输出output</span><br><span class="line"># [b, 784] =&gt; [b, 10]</span><br><span class="line">out = model(x)</span><br><span class="line"># step2, 计算mse损失</span><br><span class="line">loss = mse(out,y)</span><br><span class="line"># Step3. 计算参数的梯度 w1, w2, w3, b1, b2, b3</span><br><span class="line">grads = tape.gradient(loss, model.trainable_variables)</span><br></pre></td></tr></table></figure></li></ul>]]></content>
      
      
      <categories>
          
          <category> Tensorflow </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
            <tag> Tensorflow </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>sklearn</title>
      <link href="/2020/01/16/sklearn/"/>
      <url>/2020/01/16/sklearn/</url>
      
        <content type="html"><![CDATA[<ol><li>评估函数</li></ol><ul><li><p><a href="https://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics" target="_blank" rel="noopener">Classification metrics</a></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">from sklearn import metrics</span><br><span class="line"></span><br><span class="line">metrics.roc_auc_score(y_true,y_pre)</span><br><span class="line"></span><br><span class="line"># F1 = 2 * (precision * recall) / (precision + recall)</span><br><span class="line">metrics.f1_score(y_true,y_pre)</span><br></pre></td></tr></table></figure></li><li><p>Regression metrics</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">metrics.mean_absolute_error(y_true,y_pre)</span><br><span class="line">metrics.mean_squared_error(y_true,y_pre)</span><br></pre></td></tr></table></figure></li></ul><ol><li><a href="https://blog.csdn.net/FrankieHello/article/details/79659111" target="_blank" rel="noopener">标准化和归一化</a><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">from sklearn.preprocessing import StandardScaler</span><br><span class="line">from sklearn.preprocessing import MinMaxScaler</span><br><span class="line"></span><br><span class="line"># 标准化</span><br><span class="line">ss = StandardScaler() </span><br><span class="line">std_data = ss.fit_transform(data) </span><br><span class="line">origin_data = ss.inverse_transform(std_data) # 还原</span><br><span class="line"></span><br><span class="line"># 归一化</span><br><span class="line">mm = MinMaxScaler()</span><br><span class="line">mm_data = mm.fit_transform(data)</span><br><span class="line">origin_data = mm.inverse_transform(mm_data)</span><br></pre></td></tr></table></figure></li></ol>]]></content>
      
      
      <categories>
          
          <category> Python </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
            <tag> sklearn </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>服务器环境命令</title>
      <link href="/2020/01/06/anaconda/"/>
      <url>/2020/01/06/anaconda/</url>
      
        <content type="html"><![CDATA[<p>记录一些服务器上常用命令。</p><h1 id="Anaconda"><a href="#Anaconda" class="headerlink" title="Anaconda"></a>Anaconda</h1><ul><li>下载与安装</li></ul><ol><li>下载安装包<br><code>wget https://repo.continuum.io/archive/Anaconda3-5.3.1-Linux-x86_64.sh</code></li><li>安装<br><code>bash Anaconda3-5.3.1-Linux-x86_64.sh</code></li><li>查看安装结果<br><code>conda info</code></li><li>更新环境变量<br><code>source ~/.bashrc</code></li></ol><ul><li><p>添加环境<br><code>export PATH=&quot;/home/ry/anaconda3/bin:$PATH&quot;</code></p></li><li><p>Python 3 与 Python 2 环境转换</p></li></ul><ol><li>创建环境<br><code>conda create -n python27 python=2.7 anaconda</code></li><li>进入环境<br><code>source activate env_name</code></li><li>离开环境<br><code>source deactivate</code></li><li>列出环境<br><code>conda env list</code></li><li>删除环境<br><code>conda env remove -n env_name</code></li><li>导出环境<br><code>conda env export &gt; environment.yaml</code></li><li>加载环境<br><code>conda env create -f environment.yaml</code></li></ol><ul><li>镜像设置</li></ul><ol><li>显示所有channel<br><code>conda config --show</code></li><li>添加镜像<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/</span><br><span class="line">conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/</span><br><span class="line">conda config --set show_channel_urls yes</span><br></pre></td></tr></table></figure></li></ol><p><a href="https://zhuanlan.zhihu.com/p/260034241" target="_blank" rel="noopener">修改conda配置信息</a><br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">vim ~/.condarc</span><br></pre></td></tr></table></figure></p><p>删除 - defaults 增加 ssl_verify: false。或者直接把下方内容替换掉~/.condarc中的内容<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">channels:</span><br><span class="line">  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/</span><br><span class="line">  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/</span><br><span class="line">  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/</span><br><span class="line">  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/Paddle/</span><br><span class="line">  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/fastai/</span><br><span class="line">  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/</span><br><span class="line">  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda/</span><br><span class="line">show_channel_urls: true</span><br><span class="line">ssl_verify: false</span><br></pre></td></tr></table></figure></p><ol><li>移除镜像<br><code>conda config --remove channels https://mirrors.tuna...</code></li><li>tensorflow 1.15环境<br><code>conda create -n tf1 -c conda-forge tensorflow-gpu=1.15</code></li><li>cudnn<br><code>conda install -c anaconda cudnn</code></li></ol><ul><li><a href="https://www.cnblogs.com/huadongw/p/15193247.html" target="_blank" rel="noopener">cuda版本更改</a></li></ul><ol><li>进入/usr/local，查看linux中存在的cuda版本。</li><li><p>添加自己需要的cuda版本到环境变量， <code>vim ~/.bashrc</code>，添加以下内容到最后</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">export PATH=/usr/local/cuda-10.1/bin:$PATH</span><br><span class="line">export LD_LIBRARY_PATH=/usr/local/cuda-10.1/lib64:$LD_LIBRARY_PATH</span><br></pre></td></tr></table></figure></li><li><p>然后source一下使其生效</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">source ~/.bashrc</span><br><span class="line">nvcc -V</span><br></pre></td></tr></table></figure></li></ol><h1 id="Tmux"><a href="#Tmux" class="headerlink" title="Tmux"></a>Tmux</h1><ol><li>安装<br><code>sudo apt install tmux</code></li><li>创建会话<br><code>tmux new -s name</code></li><li>分离会话，退出当前tmux窗口，使之在后台运行<br><code>tmux detach</code> 或 <code>Ctrl+b d</code></li><li>查看所有会话<br><code>tmux ls</code> 或 <code>tmux list-session</code></li><li>接入会话<br><code>tmux a -t name</code> 或 <code>tmux attach -t name</code></li><li>杀死会话<br><code>tmux kill-session -t name</code></li><li>切换会话<br><code>tmux switch -t name</code></li><li>重命名会话<br><code>tmux rename-session -t name</code></li></ol><h1 id="VNC"><a href="#VNC" class="headerlink" title="VNC"></a>VNC</h1><ol><li>创建会话<br><code>vncserver</code></li><li>设置密码<br><code>vncpasswd</code></li><li>登录<br><code>ip:server id</code></li></ol><h1 id="ubuntu"><a href="#ubuntu" class="headerlink" title="ubuntu"></a>ubuntu</h1><ol><li><p>删除文件夹<br><code>rm -rf /filename</code></p></li><li><p>文件同步(慢)<br><code>rsync -av /source /destination</code><br><code>rm -rf /source</code></p></li><li><p>移动文件夹<br><code>mv /source/* /destination</code></p></li></ol>]]></content>
      
      
      <categories>
          
          <category> Notes </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Anaconda </tag>
            
            <tag> tmux </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Pytorch 序列</title>
      <link href="/2019/12/17/pytorch3/"/>
      <url>/2019/12/17/pytorch3/</url>
      
        <content type="html"><![CDATA[<p>参考：<a href="https://github.com/L1aoXingyu/code-of-learn-deep-learning-with-pytorch/blob/master/chapter2_PyTorch-Basics/PyTorch-introduction.ipynb">code of learn deep learning with pytroch</a></p><h1 id="循环神经网络"><a href="#循环神经网络" class="headerlink" title="循环神经网络"></a>循环神经网络</h1><h2 id="RNN"><a href="#RNN" class="headerlink" title="RNN"></a>RNN</h2><script type="math/tex; mode=display">h_t = tanh(w_{ih}*x_t+b_{ih}+w_{hh}*h_{t-1}+b_{hh})</script><ol><li><p><code>torch.nn.RNNCell()</code> 只接受序列中单步的输入，且必须传入隐藏状态</p><ul><li><code>input_size</code>: 输入的特征维度</li><li><code>hidden_size</code>: 输出的特征维度</li><li><code>num_layers</code>: 网络的层数</li><li><code>nonlinearity</code>: 非线性激活函数，默认 ‘tanh’</li><li><code>bias</code>: 是否使用偏置，默认使用</li><li><code>batch_first</code>: 输入数据的形式，默认 False，为(seq, batch, feature)，</li><li><code>dropout</code>: 是否应用 dropout</li><li><code>bidirectional</code>: 是否使用双向 rnn，默认 False<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">import torch</span><br><span class="line">from torch.autograd import Variable</span><br><span class="line">from torch import nn</span><br><span class="line"></span><br><span class="line"># 定义一个单步的 rnn</span><br><span class="line">rnn_single = nn.RNNCell(input_size=100, hidden_size=200)</span><br><span class="line"></span><br><span class="line"># 访问其中的参数</span><br><span class="line">rnn_single.weight_hh</span><br><span class="line"></span><br><span class="line"># 构造一个序列，长为 6，batch 是 5， 特征是 100</span><br><span class="line">x = Variable(torch.randn(6, 5, 100)) # 这是 rnn 的输入格式</span><br><span class="line"></span><br><span class="line"># 定义初始的记忆状态</span><br><span class="line">h_t = Variable(torch.zeros(5, 200))</span><br><span class="line"></span><br><span class="line"># 传入 rnn</span><br><span class="line">out = []</span><br><span class="line">for i in range(6): # 通过循环 6 次作用在整个序列上</span><br><span class="line">    h_t = rnn_single(x[i], h_t)</span><br><span class="line">    out.append(h_t)</span><br></pre></td></tr></table></figure></li></ul></li><li><p><code>torch.nn.RNN()</code> 可以接受一个序列的输入，默认为全0的隐藏状态，可以自己申明</p><ul><li><code>input_size</code></li><li><code>hidden_size</code></li><li><code>bias</code></li><li><code>nonlinearity</code><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">rnn_seq = nn.RNN(100, 200)</span><br><span class="line"></span><br><span class="line"># 访问其中的参数</span><br><span class="line">rnn_seq.weight_hh_l0</span><br><span class="line"></span><br><span class="line"># 使用默认的全 0 隐藏状态</span><br><span class="line">out, h_t = rnn_seq(x) </span><br><span class="line"></span><br><span class="line"># 自己定义初始的隐藏状态</span><br><span class="line">h_0 = Variable(torch.randn(1, 5, 200))</span><br><span class="line">out, h_t = rnn_seq(x, h_0)</span><br></pre></td></tr></table></figure></li></ul></li></ol><p>输出的结果均为 (seq, batch, feature)</p><h2 id="LSTM"><a href="#LSTM" class="headerlink" title="LSTM"></a>LSTM</h2><script type="math/tex; mode=display">f_t = \sigma (W_f \cdot [h_{t-1},x_t]+b_f) \\i_t = \sigma (W_i \cdot [h_{t-1},x_t]+b_i) \\\tilde{C_t} = tanh (W_C \cdot [h_{t-1},x_t]+b_C) \\C_t = f_t*C_{t-1}+i_t *\tilde{C_t} \\o_t = \sigma(w_o \cdot [h_{t-1},x_t]+b_o) \\h_t = o_t*tanh(C_t)</script><p>LSTM 与基本 RNN 一样，参数也相同，具有<code>nn.LSTMCell()</code>和<code>nn.LSTM()</code>两种形式。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">lstm_seq = nn.LSTM(50, 100, num_layers=2) # 输入维度 50，输出 100，两层</span><br><span class="line"></span><br><span class="line">lstm_seq.weight_hh_l0 # 第一层的 h_t 权重</span><br><span class="line"></span><br><span class="line">lstm_input = Variable(torch.randn(10, 3, 50)) # 序列 10，batch 是 3，输入维度 50</span><br><span class="line"></span><br><span class="line">out, (h, c) = lstm_seq(lstm_input) # 使用默认的全 0 隐藏状态</span><br><span class="line"></span><br><span class="line"># 不使用默认的隐藏状态</span><br><span class="line">h_init = Variable(torch.randn(2, 3, 100))</span><br><span class="line">c_init = Variable(torch.randn(2, 3, 100))</span><br><span class="line"></span><br><span class="line">out, (h, c) = lstm_seq(lstm_input, (h_init, c_init))</span><br></pre></td></tr></table></figure></p><h2 id="GRU"><a href="#GRU" class="headerlink" title="GRU"></a>GRU</h2><script type="math/tex; mode=display">z_t = \sigma (W_z \cdot [h_{t-1},x_t]) \\r_t = \sigma (W_r \cdot [h_{t-1},x_t]) \\\tilde{h_t} = tanh(W \cdot [r_t * h_{t-1}, x_t]) \\h_t = (1-z_t)*h_{t-1}+z_t*\tilde{h_t}</script><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">gru_seq = nn.GRU(10, 20)</span><br><span class="line">gru_input = Variable(torch.randn(3, 32, 10))</span><br><span class="line"></span><br><span class="line">out, h = gru_seq(gru_input)</span><br></pre></td></tr></table></figure><h2 id="RNN用于时间序列分析"><a href="#RNN用于时间序列分析" class="headerlink" title="RNN用于时间序列分析"></a>RNN用于时间序列分析</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br></pre></td><td class="code"><pre><span class="line">import numpy as np</span><br><span class="line">import pandas as pd</span><br><span class="line">import matplotlib.pyplot as plt</span><br><span class="line"></span><br><span class="line">import torch</span><br><span class="line">from torch import nn</span><br><span class="line">from torch.autograd import Variable</span><br><span class="line"></span><br><span class="line">data_csv = pd.read_csv(&apos;./data.csv&apos;, usecols=[1])</span><br><span class="line"></span><br><span class="line"># 数据预处理</span><br><span class="line">data_csv = data_csv.dropna()</span><br><span class="line">dataset = data_csv.values</span><br><span class="line">dataset = dataset.astype(&apos;float32&apos;)</span><br><span class="line">max_value = np.max(dataset)</span><br><span class="line">min_value = np.min(dataset)</span><br><span class="line">scalar = max_value - min_value</span><br><span class="line">dataset = list(map(lambda x: (x-min_value) / scalar, dataset))</span><br><span class="line"></span><br><span class="line"># 通过前两个月的流量来预测当月流量</span><br><span class="line">def create_dataset(dataset, look_back=2):</span><br><span class="line">    dataX, dataY = [], []</span><br><span class="line">    for i in range(len(dataset) - look_back):</span><br><span class="line">        a = dataset[i:(i + look_back)]</span><br><span class="line">        dataX.append(a)</span><br><span class="line">        dataY.append(dataset[i + look_back])</span><br><span class="line">    return np.array(dataX), np.array(dataY)</span><br><span class="line"></span><br><span class="line"># 创建好输入输出</span><br><span class="line">data_X, data_Y = create_dataset(dataset)</span><br><span class="line"></span><br><span class="line"># 划分训练集和测试集，70% 作为训练集</span><br><span class="line">train_size = int(len(data_X) * 0.7)</span><br><span class="line">test_size = len(data_X) - train_size</span><br><span class="line">train_X = data_X[:train_size]</span><br><span class="line">train_Y = data_Y[:train_size]</span><br><span class="line">test_X = data_X[train_size:]</span><br><span class="line">test_Y = data_Y[train_size:]</span><br><span class="line"></span><br><span class="line"># RNN 读入数据维度为 (seq,batch,feature)</span><br><span class="line">train_X = train_X.reshape(-1, 1, 2)</span><br><span class="line">train_Y = train_Y.reshape(-1, 1, 1)</span><br><span class="line">test_X = test_X.reshape(-1, 1, 2)</span><br><span class="line"></span><br><span class="line">train_x = torch.from_numpy(train_X)</span><br><span class="line">train_y = torch.from_numpy(train_Y)</span><br><span class="line">test_x = torch.from_numpy(test_X)</span><br><span class="line"></span><br><span class="line"># 定义模型</span><br><span class="line">class lstm_reg(nn.Module):</span><br><span class="line">    def __init__(self, input_size, hidden_size, output_size=1, num_layers=2):</span><br><span class="line">        super(lstm_reg, self).__init__()</span><br><span class="line">        </span><br><span class="line">        self.rnn = nn.LSTM(input_size, hidden_size, num_layers) # rnn</span><br><span class="line">        self.reg = nn.Linear(hidden_size, output_size) # 回归</span><br><span class="line">        </span><br><span class="line">    def forward(self, x):</span><br><span class="line">        x, _ = self.rnn(x) # (seq, batch, hidden)</span><br><span class="line">        s, b, h = x.shape</span><br><span class="line">        x = x.view(s*b, h) # 转换成线性层的输入格式</span><br><span class="line">        x = self.reg(x)</span><br><span class="line">        x = x.view(s, b, -1)</span><br><span class="line">        return x</span><br><span class="line"># 输入维度为2，隐层为4</span><br><span class="line">net = lstm_reg(2, 4)</span><br><span class="line"></span><br><span class="line">criterion = nn.MSELoss()</span><br><span class="line">optimizer = torch.optim.Adam(net.parameters(), lr=1e-2)</span><br><span class="line"></span><br><span class="line"># 开始训练</span><br><span class="line">for e in range(1000):</span><br><span class="line">    var_x = Variable(train_x)</span><br><span class="line">    var_y = Variable(train_y)</span><br><span class="line">    # 前向传播</span><br><span class="line">    out = net(var_x)</span><br><span class="line">    loss = criterion(out, var_y)</span><br><span class="line">    # 反向传播</span><br><span class="line">    optimizer.zero_grad()</span><br><span class="line">    loss.backward()</span><br><span class="line">    optimizer.step()</span><br><span class="line">    if (e + 1) % 100 == 0: # 每 100 次输出结果</span><br><span class="line">        print(&apos;Epoch: &#123;&#125;, Loss: &#123;:.5f&#125;&apos;.format(e + 1, loss.data[0]))</span><br><span class="line"></span><br><span class="line"># 预测结果</span><br><span class="line">net = net.eval() # 转换成测试模式</span><br><span class="line"></span><br><span class="line">data_X = data_X.reshape(-1, 1, 2)</span><br><span class="line">data_X = torch.from_numpy(data_X)</span><br><span class="line">var_data = Variable(data_X)</span><br><span class="line">pred_test = net(var_data) # 测试集的预测结果</span><br><span class="line"></span><br><span class="line"># 改变输出的格式，view=reshape</span><br><span class="line">pred_test = pred_test.view(-1).data.numpy()</span><br><span class="line"></span><br><span class="line"># 画出实际结果和预测的结果</span><br><span class="line">plt.plot(pred_test, &apos;r&apos;, label=&apos;prediction&apos;)</span><br><span class="line">plt.plot(dataset, &apos;b&apos;, label=&apos;real&apos;)</span><br><span class="line">plt.legend(loc=&apos;best&apos;)</span><br></pre></td></tr></table></figure><h1 id="Embedding"><a href="#Embedding" class="headerlink" title="Embedding"></a>Embedding</h1><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line">import torch</span><br><span class="line">from torch import nn</span><br><span class="line">from torch.autograd import Variable</span><br><span class="line"></span><br><span class="line"># 定义词嵌入</span><br><span class="line">embeds = nn.Embedding(2, 5) # 2 个单词，维度 5</span><br><span class="line"></span><br><span class="line"># 得到词嵌入矩阵</span><br><span class="line">embeds.weight</span><br><span class="line"></span><br><span class="line"># 直接手动修改词嵌入的值</span><br><span class="line">embeds.weight.data = torch.ones(2, 5)</span><br><span class="line"></span><br><span class="line"># 访问第 50 个词的词向量</span><br><span class="line">embeds = nn.Embedding(100, 10)</span><br><span class="line">single_word_embed = embeds(Variable(torch.LongTensor([50])))</span><br></pre></td></tr></table></figure><h1 id="N-Gram"><a href="#N-Gram" class="headerlink" title="N-Gram"></a>N-Gram</h1><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br></pre></td><td class="code"><pre><span class="line">CONTEXT_SIZE = 2 # 依据的单词数</span><br><span class="line">EMBEDDING_DIM = 10 # 词向量的维度</span><br><span class="line"># 我们使用莎士比亚的诗</span><br><span class="line">test_sentence = &quot;&quot;&quot;When forty winters shall besiege thy brow,</span><br><span class="line">And dig deep trenches in thy beauty&apos;s field,</span><br><span class="line">Thy youth&apos;s proud livery so gazed on now,</span><br><span class="line">Will be a totter&apos;d weed of small worth held:</span><br><span class="line">Then being asked, where all thy beauty lies,</span><br><span class="line">Where all the treasure of thy lusty days;</span><br><span class="line">To say, within thine own deep sunken eyes,</span><br><span class="line">Were an all-eating shame, and thriftless praise.</span><br><span class="line">How much more praise deserv&apos;d thy beauty&apos;s use,</span><br><span class="line">If thou couldst answer &apos;This fair child of mine</span><br><span class="line">Shall sum my count, and make my old excuse,&apos;</span><br><span class="line">Proving his beauty by succession thine!</span><br><span class="line">This were to be new made when thou art old,</span><br><span class="line">And see thy blood warm when thou feel&apos;st it cold.&quot;&quot;&quot;.split()</span><br><span class="line"></span><br><span class="line"># 创建数据集</span><br><span class="line">trigram = [((test_sentence[i], test_sentence[i+1]), test_sentence[i+2]) </span><br><span class="line">            for i in range(len(test_sentence)-2)]</span><br><span class="line"></span><br><span class="line"># 建立每个词与数字的编码，据此构建词嵌入</span><br><span class="line">vocb = set(test_sentence) # 使用 set 将重复的元素去掉</span><br><span class="line">word_to_idx = &#123;word: i for i, word in enumerate(vocb)&#125;</span><br><span class="line">idx_to_word = &#123;word_to_idx[word]: word for word in word_to_idx&#125;</span><br><span class="line"></span><br><span class="line"># 定义模型</span><br><span class="line">class n_gram(nn.Module):</span><br><span class="line">    def __init__(self, vocab_size, context_size=CONTEXT_SIZE, n_dim=EMBEDDING_DIM):</span><br><span class="line">        super(n_gram, self).__init__()</span><br><span class="line">        </span><br><span class="line">        self.embed = nn.Embedding(vocab_size, n_dim)</span><br><span class="line">        self.classify = nn.Sequential(</span><br><span class="line">            nn.Linear(context_size * n_dim, 128),</span><br><span class="line">            nn.ReLU(True),</span><br><span class="line">            nn.Linear(128, vocab_size)</span><br><span class="line">        )</span><br><span class="line">        </span><br><span class="line">    def forward(self, x):</span><br><span class="line">        voc_embed = self.embed(x) # 得到词嵌入</span><br><span class="line">        voc_embed = voc_embed.view(1, -1) # 将两个词向量拼在一起</span><br><span class="line">        out = self.classify(voc_embed)</span><br><span class="line">        return out</span><br><span class="line"></span><br><span class="line">net = n_gram(len(word_to_idx))</span><br><span class="line"></span><br><span class="line">criterion = nn.CrossEntropyLoss()</span><br><span class="line">optimizer = torch.optim.SGD(net.parameters(), lr=1e-2, weight_decay=1e-5)</span><br><span class="line"></span><br><span class="line">for e in range(100):</span><br><span class="line">    train_loss = 0</span><br><span class="line">    for word, label in trigram: # 使用前 100 个作为训练集</span><br><span class="line">        word = Variable(torch.LongTensor([word_to_idx[i] for i in word])) # 将两个词作为输入</span><br><span class="line">        label = Variable(torch.LongTensor([word_to_idx[label]]))</span><br><span class="line">        # 前向传播</span><br><span class="line">        out = net(word)</span><br><span class="line">        loss = criterion(out, label)</span><br><span class="line">        train_loss += loss.data</span><br><span class="line">        # 反向传播</span><br><span class="line">        optimizer.zero_grad()</span><br><span class="line">        loss.backward()</span><br><span class="line">        optimizer.step()</span><br><span class="line">    if (e + 1) % 20 == 0:</span><br><span class="line">        print(&apos;epoch: &#123;&#125;, Loss: &#123;:.6f&#125;&apos;.format(e + 1, train_loss / len(trigram)))</span><br><span class="line"></span><br><span class="line">net = net.eval()</span><br><span class="line"></span><br><span class="line"># 测试一下结果</span><br><span class="line">word, label = trigram[19]</span><br><span class="line">print(&apos;input: &#123;&#125;&apos;.format(word))</span><br><span class="line">print(&apos;label: &#123;&#125;&apos;.format(label))</span><br><span class="line">print()</span><br><span class="line">word = Variable(torch.LongTensor([word_to_idx[i] for i in word]))</span><br><span class="line">out = net(word)</span><br><span class="line"></span><br><span class="line">pred_label_idx = out.max(1)[1].data.numpy()[0]</span><br><span class="line">print(pred_label_idx)</span><br><span class="line">predict_word = idx_to_word[pred_label_idx]</span><br><span class="line">print(&apos;real word is &#123;&#125;, predicted word is &#123;&#125;&apos;.format(label, predict_word))</span><br></pre></td></tr></table></figure><h1 id="LSTM-做词性预测"><a href="#LSTM-做词性预测" class="headerlink" title="LSTM 做词性预测"></a>LSTM 做词性预测</h1><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br></pre></td><td class="code"><pre><span class="line">import torch</span><br><span class="line">from torch import nn</span><br><span class="line">from torch.autograd import Variable</span><br><span class="line"></span><br><span class="line">training_data = [(&quot;The dog ate the apple&quot;.split(),</span><br><span class="line">                  [&quot;DET&quot;, &quot;NN&quot;, &quot;V&quot;, &quot;DET&quot;, &quot;NN&quot;]),</span><br><span class="line">                 (&quot;Everybody read that book&quot;.split(), </span><br><span class="line">                  [&quot;NN&quot;, &quot;V&quot;, &quot;DET&quot;, &quot;NN&quot;])]</span><br><span class="line"></span><br><span class="line"># 对单词和标签编码</span><br><span class="line">word_to_idx = &#123;&#125;</span><br><span class="line">tag_to_idx = &#123;&#125;</span><br><span class="line">for context, tag in training_data:</span><br><span class="line">    for word in context:</span><br><span class="line">        if word.lower() not in word_to_idx:</span><br><span class="line">            word_to_idx[word.lower()] = len(word_to_idx)</span><br><span class="line">    for label in tag:</span><br><span class="line">        if label.lower() not in tag_to_idx:</span><br><span class="line">            tag_to_idx[label.lower()] = len(tag_to_idx)</span><br><span class="line"></span><br><span class="line"># 对字母编码</span><br><span class="line">alphabet = &apos;abcdefghijklmnopqrstuvwxyz&apos;</span><br><span class="line">char_to_idx = &#123;&#125;</span><br><span class="line">for i in range(len(alphabet)):</span><br><span class="line">    char_to_idx[alphabet[i]] = i</span><br><span class="line"></span><br><span class="line"># 构建训练数据</span><br><span class="line">def make_sequence(x, dic): # 字符编码</span><br><span class="line">    idx = [dic[i.lower()] for i in x]</span><br><span class="line">    idx = torch.LongTensor(idx)</span><br><span class="line">    return idx</span><br><span class="line"></span><br><span class="line"># 构建单个字符的 lstm 模型</span><br><span class="line">class char_lstm(nn.Module):</span><br><span class="line">    def __init__(self, n_char, char_dim, char_hidden):</span><br><span class="line">        super(char_lstm, self).__init__()</span><br><span class="line">        </span><br><span class="line">        self.char_embed = nn.Embedding(n_char, char_dim)</span><br><span class="line">        self.lstm = nn.LSTM(char_dim, char_hidden)</span><br><span class="line">        </span><br><span class="line">    def forward(self, x):</span><br><span class="line">        x = self.char_embed(x)</span><br><span class="line">        out, _ = self.lstm(x)</span><br><span class="line">        return out[-1] # (batch, hidden)</span><br><span class="line"></span><br><span class="line"># 构建词性分类的 lstm 模型</span><br><span class="line">class lstm_tagger(nn.Module):</span><br><span class="line">    def __init__(self, n_word, n_char, char_dim, word_dim, </span><br><span class="line">                 char_hidden, word_hidden, n_tag):</span><br><span class="line">        super(lstm_tagger, self).__init__()</span><br><span class="line">        self.word_embed = nn.Embedding(n_word, word_dim)</span><br><span class="line">        self.char_lstm = char_lstm(n_char, char_dim, char_hidden)</span><br><span class="line">        self.word_lstm = nn.LSTM(word_dim + char_hidden, word_hidden)</span><br><span class="line">        self.classify = nn.Linear(word_hidden, n_tag)</span><br><span class="line">        </span><br><span class="line">    def forward(self, x, word):</span><br><span class="line">        char = []</span><br><span class="line">        for w in word: # 对于每个单词做字符的 lstm</span><br><span class="line">            char_list = make_sequence(w, char_to_idx)</span><br><span class="line">            char_list = char_list.unsqueeze(1) # (seq, batch, feature) 满足 lstm 输入条件</span><br><span class="line">            char_infor = self.char_lstm(Variable(char_list)) # (batch, char_hidden)</span><br><span class="line">            char.append(char_infor)</span><br><span class="line">        char = torch.stack(char, dim=0) # (seq, batch, feature)</span><br><span class="line">        </span><br><span class="line">        x = self.word_embed(x) # (batch, seq, word_dim)</span><br><span class="line">        x = x.permute(1, 0, 2) # 改变顺序</span><br><span class="line">        x = torch.cat((x, char), dim=2) # 沿着特征通道将每个词的词嵌入和字符 lstm 输出的结果拼接在一起</span><br><span class="line">        x, _ = self.word_lstm(x)</span><br><span class="line">        </span><br><span class="line">        s, b, h = x.shape</span><br><span class="line">        x = x.view(-1, h) # 重新 reshape 进行分类线性层</span><br><span class="line">        out = self.classify(x)</span><br><span class="line">        return out</span><br><span class="line"></span><br><span class="line">net = lstm_tagger(len(word_to_idx), len(char_to_idx), 10, 100, 50, 128, len(tag_to_idx))</span><br><span class="line">criterion = nn.CrossEntropyLoss()</span><br><span class="line">optimizer = torch.optim.SGD(net.parameters(), lr=1e-2)</span><br><span class="line"></span><br><span class="line"># 开始训练</span><br><span class="line">for e in range(300):</span><br><span class="line">    train_loss = 0</span><br><span class="line">    for word, tag in training_data:</span><br><span class="line">        word_list = make_sequence(word, word_to_idx).unsqueeze(0) # 添加第一维 batch</span><br><span class="line">        tag = make_sequence(tag, tag_to_idx)</span><br><span class="line">        word_list = Variable(word_list)</span><br><span class="line">        tag = Variable(tag)</span><br><span class="line">        # 前向传播</span><br><span class="line">        out = net(word_list, word)</span><br><span class="line">        loss = criterion(out, tag)</span><br><span class="line">        train_loss += loss.data[0]</span><br><span class="line">        # 反向传播</span><br><span class="line">        optimizer.zero_grad()</span><br><span class="line">        loss.backward()</span><br><span class="line">        optimizer.step()</span><br><span class="line">    if (e + 1) % 50 == 0:</span><br><span class="line">        print(&apos;Epoch: &#123;&#125;, Loss: &#123;:.5f&#125;&apos;.format(e + 1, train_loss / len(training_data)))</span><br><span class="line"></span><br><span class="line"># 预测</span><br><span class="line">net = net.eval()</span><br><span class="line"></span><br><span class="line">test_sent = &apos;Everybody ate the apple&apos;</span><br><span class="line">test = make_sequence(test_sent.split(), word_to_idx).unsqueeze(0)</span><br><span class="line">out = net(Variable(test), test_sent.split())</span><br><span class="line"></span><br><span class="line">print(tag_to_idx)</span><br></pre></td></tr></table></figure>]]></content>
      
      
      <categories>
          
          <category> Pytorch </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
            <tag> Pytorch </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Pytorch 模型</title>
      <link href="/2019/12/17/pytorch2/"/>
      <url>/2019/12/17/pytorch2/</url>
      
        <content type="html"><![CDATA[<p>参考：<a href="https://github.com/L1aoXingyu/code-of-learn-deep-learning-with-pytorch/blob/master/chapter2_PyTorch-Basics/PyTorch-introduction.ipynb">code of learn deep learning with pytroch</a></p><h2 id="Sequential"><a href="#Sequential" class="headerlink" title="Sequential"></a>Sequential</h2><p>Sequential 构建序列化模块。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">seq_net = nn.Sequential(</span><br><span class="line">nn.Linear(2,4),</span><br><span class="line">nn.Tanh(),</span><br><span class="line">nn.Linear(4,1)</span><br><span class="line">)</span><br></pre></td></tr></table></figure></p><ul><li><p>序列模块可以通过索引访问每一层</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">seq_net[0] # 第一层</span><br><span class="line">[out]: Linear(in_features=2, out_features=4)</span><br></pre></td></tr></table></figure></li><li><p>打印权重</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">w0 = seq_net[0].weight</span><br><span class="line">print(w0)</span><br></pre></td></tr></table></figure></li><li><p>训练模型：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"># 通过 parameters 可以取得模型的参数</span><br><span class="line">param = seq_net.parameters()</span><br><span class="line"></span><br><span class="line"># 定义优化器</span><br><span class="line">optim = torch.optim.SGD(param, 1.)</span><br><span class="line"></span><br><span class="line"># 训练 10000 次</span><br><span class="line">for e in range(10000):</span><br><span class="line">    out = seq_net(Variable(x))</span><br><span class="line">    loss = criterion(out, Variable(y))</span><br><span class="line">    optim.zero_grad()</span><br><span class="line">    loss.backward()</span><br><span class="line">    optim.step()</span><br><span class="line">    if (e + 1) % 1000 == 0:</span><br><span class="line">        print(&apos;epoch: &#123;&#125;, loss: &#123;&#125;&apos;.format(e+1, loss.data))</span><br></pre></td></tr></table></figure></li><li><p>保存参数和模型</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"># 将参数和模型保存在一起</span><br><span class="line">torch.save(seq_net, &apos;save_seq_net.pth&apos;)</span><br><span class="line"></span><br><span class="line"># 读取保存的模型</span><br><span class="line">seq_net1 = torch.load(&apos;save_seq_net.pth&apos;)</span><br></pre></td></tr></table></figure></li><li><p>保存参数</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"># 保存模型参数</span><br><span class="line">torch.save(seq_net.state_dict(), &apos;save_seq_net_params.pth&apos;)</span><br><span class="line"></span><br><span class="line"># 先定义模型，再读取参数</span><br><span class="line">seq_net2 = nn.Sequential(</span><br><span class="line">    nn.Linear(2, 4),</span><br><span class="line">    nn.Tanh(),</span><br><span class="line">    nn.Linear(4, 1)</span><br><span class="line">)</span><br><span class="line"></span><br><span class="line">seq_net2.load_state_dict(torch.load(&apos;save_seq_net_params.pth&apos;))</span><br></pre></td></tr></table></figure></li></ul><h2 id="Module"><a href="#Module" class="headerlink" title="Module"></a>Module</h2><p>更加灵活的模型定义方式。使用 Module 的模板：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">class 网络名字(nn.Module):</span><br><span class="line">    def __init__(self, 一些定义的参数):</span><br><span class="line">        super(网络名字, self).__init__()</span><br><span class="line">        self.layer1 = nn.Linear(num_input, num_hidden)</span><br><span class="line">        self.layer2 = nn.Sequential(...)</span><br><span class="line">        ...</span><br><span class="line"></span><br><span class="line">        定义需要用的网络层</span><br><span class="line"></span><br><span class="line">    def forward(self, x): # 定义前向传播</span><br><span class="line">        x1 = self.layer1(x)</span><br><span class="line">        x2 = self.layer2(x)</span><br><span class="line">        x = x1 + x2</span><br><span class="line">        ...</span><br><span class="line">        return x</span><br></pre></td></tr></table></figure></p><ul><li><p>实现上述神经网络：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">class module_net(nn.Module):</span><br><span class="line">    def __init__(self, num_input, num_hidden, num_output):</span><br><span class="line">        super(module_net, self).__init__()</span><br><span class="line">        self.layer1 = nn.Linear(num_input, num_hidden)</span><br><span class="line">        </span><br><span class="line">        self.layer2 = nn.Tanh()</span><br><span class="line">        </span><br><span class="line">        self.layer3 = nn.Linear(num_hidden, num_output)</span><br><span class="line">        </span><br><span class="line">    def forward(self, x):</span><br><span class="line">        x = self.layer1(x)</span><br><span class="line">        x = self.layer2(x)</span><br><span class="line">        x = self.layer3(x)</span><br><span class="line">        return x</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">mo_net = module_net(2, 4, 1)</span><br></pre></td></tr></table></figure></li><li><p>访问模型中的某层可以直接通过名字</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"># 第一层</span><br><span class="line">l1 = mo_net.layer1</span><br></pre></td></tr></table></figure></li><li><p>打印权值</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(l1.weight)</span><br></pre></td></tr></table></figure></li><li><p>训练模型</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"># 定义优化器</span><br><span class="line">optim = torch.optim.SGD(mo_net.parameters(), 1.)</span><br><span class="line"></span><br><span class="line"># 训练1000次</span><br><span class="line">for e in range(10000):</span><br><span class="line">    out = mo_net(Variable(x))</span><br><span class="line">    loss = criterion(out, Variable(y))</span><br><span class="line">    optim.zero_grad()</span><br><span class="line">    loss.backward()</span><br><span class="line">    optim.step()</span><br><span class="line">    if (e + 1) % 1000 == 0:</span><br><span class="line">        print(&apos;epoch: &#123;&#125;, loss: &#123;&#125;&apos;.format(e+1, loss.data[0]))</span><br><span class="line"></span><br><span class="line"># 保存模型</span><br><span class="line">torch.save(mo_net.state_dict(), &apos;module_net.pth&apos;)</span><br></pre></td></tr></table></figure></li><li><p>下载 MNIST 数据</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">def data_tf(x):</span><br><span class="line">    x = np.array(x, dtype=&apos;float32&apos;) / 255</span><br><span class="line">    x = (x - 0.5) / 0.5 # 标准化，这个技巧之后会讲到</span><br><span class="line">    x = x.reshape((-1,)) # 拉平</span><br><span class="line">    x = torch.from_numpy(x)</span><br><span class="line">    return x</span><br><span class="line"></span><br><span class="line">train_set = mnist.MNIST(&apos;./data&apos;, train=True, transform=data_tf, download=True) # 重新载入数据集，申明定义的数据变换</span><br><span class="line">test_set = mnist.MNIST(&apos;./data&apos;, train=False, transform=data_tf, download=True)</span><br></pre></td></tr></table></figure></li><li><p>使用 DataLoader 定义一个数据迭代器</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">from torch.utils.data import DataLoader</span><br><span class="line">train_data = DataLoader(train_set, batch_size=64, shuffle=True)</span><br><span class="line">test_data = DataLoader(test_set, batch_size=128, shuffle=False)</span><br><span class="line"></span><br><span class="line"># iter()获取迭代对象的迭代器，next()获取下一条数据</span><br><span class="line">a, a_label = next(iter(train_data))</span><br></pre></td></tr></table></figure></li><li><p>绘制图像</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">plt.title(&apos;train loss&apos;)</span><br><span class="line">plt.plot(np.arange(len(losses)), losses)</span><br></pre></td></tr></table></figure></li></ul><h2 id="参数初始化"><a href="#参数初始化" class="headerlink" title="参数初始化"></a>参数初始化</h2><ul><li><p>可以通过<code>.weigth</code>和<code>.bias</code>访问网络的权值，并通过<code>.data</code>访问其数值，并替换：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"># 定义一个 Sequential 模型</span><br><span class="line">net1 = nn.Sequential(</span><br><span class="line">    nn.Linear(30, 40),</span><br><span class="line">    nn.ReLU(),</span><br><span class="line">    nn.Linear(40, 50),</span><br><span class="line">    nn.ReLU(),</span><br><span class="line">    nn.Linear(50, 10)</span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"># 定义一个 Tensor 直接对其进行替换</span><br><span class="line">net1[0].weight.data = torch.from_numpy(np.random.uniform(3, 5, size=(40, 30)))</span><br><span class="line"></span><br><span class="line"># 模型中相同类型的层需要相同初始化方式</span><br><span class="line">for layer in net1:</span><br><span class="line">    if isinstance(layer, nn.Linear): # 判断是否是线性层</span><br><span class="line">        param_shape = layer.weight.shape</span><br><span class="line">        layer.weight.data = torch.from_numpy(np.random.normal(0, 0.5, size=param_shape)) </span><br><span class="line">        # 定义为均值为 0，方差为 0.5 的正态分布</span><br></pre></td></tr></table></figure></li><li><p>对于 Module 的参数化，可以直接像 Sequential 一样对其 Tensor 进行重新定义。如果用循环方式，需要介绍两个属性，children 和 modules, children 只访问到模型定义中的第一层，modules 会访问到最后的结构：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br></pre></td><td class="code"><pre><span class="line">class sim_net(nn.Module):</span><br><span class="line">    def __init__(self):</span><br><span class="line">        super(sim_net, self).__init__()</span><br><span class="line">        self.l1 = nn.Sequential(</span><br><span class="line">            nn.Linear(30, 40),</span><br><span class="line">            nn.ReLU()</span><br><span class="line">        )</span><br><span class="line">        </span><br><span class="line">        self.l1[0].weight.data = torch.randn(40, 30) # 直接对某一层初始化</span><br><span class="line">        </span><br><span class="line">        self.l2 = nn.Sequential(</span><br><span class="line">            nn.Linear(40, 50),</span><br><span class="line">            nn.ReLU()</span><br><span class="line">        )</span><br><span class="line">        </span><br><span class="line">        self.l3 = nn.Sequential(</span><br><span class="line">            nn.Linear(50, 10),</span><br><span class="line">            nn.ReLU()</span><br><span class="line">        )</span><br><span class="line">    </span><br><span class="line">    def forward(self, x):</span><br><span class="line">        x = self.l1(x)</span><br><span class="line">        x =self.l2(x)</span><br><span class="line">        x = self.l3(x)</span><br><span class="line">        return x</span><br><span class="line"></span><br><span class="line">net2 = sim_net()</span><br><span class="line"></span><br><span class="line"># 访问 children</span><br><span class="line">for i in net2.children():</span><br><span class="line">    print(i)</span><br><span class="line"></span><br><span class="line"># 访问 modules</span><br><span class="line">for i in net2.modules():</span><br><span class="line">    print(i)</span><br><span class="line"></span><br><span class="line"># 迭代初始化</span><br><span class="line">for layer in net2.modules():</span><br><span class="line">    if isinstance(layer, nn.Linear):</span><br><span class="line">        param_shape = layer.weight.shape</span><br><span class="line">        layer.weight.data = torch.from_numpy(np.random.normal(0, 0.5, size=param_shape))</span><br></pre></td></tr></table></figure></li><li><p>torch.nn.init</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">from torch.nn import init</span><br><span class="line"></span><br><span class="line">init.xavier_uniform(net1[0].weight)</span><br></pre></td></tr></table></figure></li></ul><h2 id="优化算法"><a href="#优化算法" class="headerlink" title="优化算法"></a>优化算法</h2><ul><li><p>随机梯度下降</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">def sgd_update(parameters, lr):</span><br><span class="line">    for param in parameters:</span><br><span class="line">        param.data = param.data - lr * param.grad.data</span><br><span class="line"></span><br><span class="line">optimzier = torch.optim.SGD(net.parameters(), 1e-2)</span><br></pre></td></tr></table></figure></li><li><p>动量法</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">def sgd_momentum(parameters, vs, lr, gamma):</span><br><span class="line">    for param, v in zip(parameters, vs):</span><br><span class="line">        v[:] = gamma * v + lr * param.grad.data</span><br><span class="line">        param.data = param.data - v</span><br><span class="line"></span><br><span class="line">optimizer = torch.optim.SGD(net.parameters(), lr=1e-2, momentum=0.9) # 加动量</span><br></pre></td></tr></table></figure></li><li><p>Adagrad</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">def sgd_adagrad(parameters, sqrs, lr):</span><br><span class="line">    eps = 1e-10</span><br><span class="line">    for param, sqr in zip(parameters, sqrs):</span><br><span class="line">        sqr[:] = sqr + param.grad.data ** 2</span><br><span class="line">        div = lr / torch.sqrt(sqr + eps) * param.grad.data</span><br><span class="line">        param.data = param.data - div</span><br><span class="line"></span><br><span class="line">optimizer = torch.optim.Adagrad(net.parameters(), lr=1e-2)</span><br></pre></td></tr></table></figure></li><li><p>RMSProp</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">def rmsprop(parameters, sqrs, lr, alpha):</span><br><span class="line">    eps = 1e-10</span><br><span class="line">    for param, sqr in zip(parameters, sqrs):</span><br><span class="line">        sqr[:] = alpha * sqr + (1 - alpha) * param.grad.data ** 2</span><br><span class="line">        div = lr / torch.sqrt(sqr + eps) * param.grad.data</span><br><span class="line">        param.data = param.data - div</span><br><span class="line"></span><br><span class="line">optimizer = torch.optim.RMSprop(net.parameters(), lr=1e-3, alpha=0.9)</span><br></pre></td></tr></table></figure></li><li><p>Adadelta</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">def adadelta(parameters, sqrs, deltas, rho):</span><br><span class="line">    eps = 1e-6</span><br><span class="line">    for param, sqr, delta in zip(parameters, sqrs, deltas):</span><br><span class="line">        sqr[:] = rho * sqr + (1 - rho) * param.grad.data ** 2</span><br><span class="line">        cur_delta = torch.sqrt(delta + eps) / torch.sqrt(sqr + eps) * param.grad.data</span><br><span class="line">        delta[:] = rho * delta + (1 - rho) * cur_delta ** 2</span><br><span class="line">        param.data = param.data - cur_delta</span><br><span class="line"></span><br><span class="line">optimizer = torch.optim.Adadelta(net.parameters(), rho=0.9)</span><br></pre></td></tr></table></figure></li><li><p>Adam</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">def adam(parameters, vs, sqrs, lr, t, beta1=0.9, beta2=0.999):</span><br><span class="line">    eps = 1e-8</span><br><span class="line">    for param, v, sqr in zip(parameters, vs, sqrs):</span><br><span class="line">        v[:] = beta1 * v + (1 - beta1) * param.grad.data</span><br><span class="line">        sqr[:] = beta2 * sqr + (1 - beta2) * param.grad.data ** 2</span><br><span class="line">        v_hat = v / (1 - beta1 ** t)</span><br><span class="line">        s_hat = sqr / (1 - beta2 ** t)</span><br><span class="line">        param.data = param.data - lr * v_hat / torch.sqrt(s_hat + eps)</span><br><span class="line"></span><br><span class="line">optimizer = torch.optim.Adam(net.parameters(), lr=1e-3)</span><br></pre></td></tr></table></figure></li></ul>]]></content>
      
      
      <categories>
          
          <category> Pytorch </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
            <tag> Pytorch </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Pytorch 基本概念</title>
      <link href="/2019/12/09/pytorch/"/>
      <url>/2019/12/09/pytorch/</url>
      
        <content type="html"><![CDATA[<p>参考：<a href="https://pytorch.org/" target="_blank" rel="noopener">官网</a>, <a href="https://github.com/L1aoXingyu/code-of-learn-deep-learning-with-pytorch/blob/master/chapter2_PyTorch-Basics/PyTorch-introduction.ipynb">深度学习入门之Pytorch</a>，<a href="https://pytorch-cn.readthedocs.io/zh/latest/notes/autograd/" target="_blank" rel="noopener">PyTorch中文文档</a></p><h2 id="Tensor-amp-Variable"><a href="#Tensor-amp-Variable" class="headerlink" title="Tensor &amp; Variable"></a>Tensor &amp; Variable</h2><ol><li><p>Pytorch 是一个拥有强力GPU加速的张量和动态构建网络的库，其主要构建是张量。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">import torch</span><br><span class="line">import numpy as np</span><br><span class="line"></span><br><span class="line"># 创建 numpy ndarray</span><br><span class="line">numpy_tensor = np.random.randn(10,20)</span><br><span class="line"></span><br><span class="line"># 创建 pytorch tensor</span><br><span class="line">pytorch_tensor = torch.randn(3,2)</span><br><span class="line"></span><br><span class="line"># 将 ndarray 转换到 tensor 上</span><br><span class="line">pytorch_tensor1 = torch.Tensor(numpy_tensor)</span><br><span class="line">pytorch_tensor2 = torch.from_numpy(numpy_tensor)</span><br><span class="line"></span><br><span class="line"># 将 tensor 转换为 ndarray</span><br><span class="line"># cpu</span><br><span class="line">numpy_array = pytorch_tensor1.numpy()</span><br><span class="line"># gpu</span><br><span class="line">numpy_array = pytorch_tensor1.cpu().numpy()</span><br></pre></td></tr></table></figure></li><li><p>将 tensor 放到 GPU 上</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"># 第一种方式：定义 cuda 数据类型</span><br><span class="line">dtype = torch.cuda.FloatTensor # 定义默认 GPU 的 数据类型</span><br><span class="line">gpu_tensor = torch.randn(10, 20).type(dtype)</span><br><span class="line"></span><br><span class="line"># 第二种方式：</span><br><span class="line">gpu_tensor = torch.randn(10, 20).cuda(0) # 将 tensor 放到第一个 GPU 上</span><br><span class="line">gpu_tensor = torch.randn(10, 20).cuda(1) # 将 tensor 放到第二个 GPU 上</span><br></pre></td></tr></table></figure></li></ol><p>使用第一种方式将 tensor 放到 GPU 上的时候会将数据类型转换成定义的类型，而是用第二种方式能够直接将 tensor 放到 GPU 上，类型跟之前保持一致</p><ol><li><p>将 tensor 放回 CPU</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">cpu_tensor = gpu_tensor.cpu()</span><br></pre></td></tr></table></figure></li><li><p>访问 tensor 的属性</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"># 大小</span><br><span class="line">pytorch_tensor1.shape</span><br><span class="line">pytorch_tensor1.size()</span><br><span class="line"></span><br><span class="line"># 类型</span><br><span class="line">pytorch_tensor1.type()</span><br><span class="line">pytorch_tensor1.type(torch.DoubleTensor) #转换为float64</span><br><span class="line"></span><br><span class="line"># 维度</span><br><span class="line">pytorch_tensor1.dim()</span><br><span class="line"></span><br><span class="line"># 所有元素个数</span><br><span class="line">pytorch_tensor1.numel()</span><br></pre></td></tr></table></figure></li><li><p>Tensor 操作</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br></pre></td><td class="code"><pre><span class="line">x = torch.ones(2,2) # float tensor</span><br><span class="line"></span><br><span class="line">x = x.long() # 转换为整型</span><br><span class="line">x = x.float() # 转换为 float</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">x = torch.randn(4,3) # 随机矩阵</span><br><span class="line"></span><br><span class="line"># 沿行取最大值，返回每一行最大值及下标</span><br><span class="line">max_value, max_index = torch.max(x, dim=1) </span><br><span class="line"></span><br><span class="line"># 沿行求和</span><br><span class="line">sum_x = torch.sum(x, dim=1)</span><br><span class="line"></span><br><span class="line"># 增加维度</span><br><span class="line">x = x.unsqueeze(0) # 在第一维加</span><br><span class="line"></span><br><span class="line"># 减少维度</span><br><span class="line">x = x.squeeze(0) # 减少第一维</span><br><span class="line">x = x.squeeze() # 将 tensor 中所有一维去掉</span><br><span class="line"></span><br><span class="line"># 维度交换</span><br><span class="line">x = x.permute(1,0,2) # 重新排列 tensor 的维度</span><br><span class="line">x = x.transpose(0,2) # 交换 tensor 中的两个维度</span><br><span class="line"></span><br><span class="line"># 使用 view 对 tensor 进行 reshape</span><br><span class="line">x = torch.randn(3,4,5)</span><br><span class="line">x = x.view(-1,5) # -1 表示任意大小，5表示第二维变成5</span><br><span class="line"></span><br><span class="line"># 两个 tensor 求和</span><br><span class="line">x = torch.randn(3,4)</span><br><span class="line">y = torch.randn(3,4)</span><br><span class="line"></span><br><span class="line">z = x+y</span><br><span class="line">z = torch.add(x,y)</span><br><span class="line"></span><br><span class="line"># inplace 操作，在操作的符号后加_</span><br><span class="line">x.unsqueeze_(0)</span><br><span class="line">x.transpose_(1,0)</span><br><span class="line">x.add_(y)</span><br></pre></td></tr></table></figure></li><li><p>Variable<br>Variable 是对 tensor 的封装，包含三个属性：<code>.data</code> tensor 本身，<code>.grad</code> tensor 的梯度，<code>.grad_fn</code> variable 的获得方式。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">from torch.autograd import Variable</span><br><span class="line"></span><br><span class="line">x_tensor = torch.randn(10, 5)</span><br><span class="line">y_tensor = torch.randn(10, 5)</span><br><span class="line"></span><br><span class="line"># 将 tensor 变成 Variable</span><br><span class="line">x = Variable(x_tensor, requires_grad=True) # 默认 Variable 是不需要求梯度</span><br><span class="line">y = Variable(y_tensor, requires_grad=True)</span><br><span class="line"></span><br><span class="line">z = torch.sum(x + y)</span><br><span class="line"></span><br><span class="line">print(z.data)</span><br><span class="line">print(z.grad_fn)</span><br><span class="line"></span><br><span class="line"># 求 x 和 y 的梯度</span><br><span class="line">z.backward()</span><br><span class="line"></span><br><span class="line">print(x.grad)</span><br><span class="line">print(y.grad)</span><br></pre></td></tr></table></figure></li></ol><h2 id="自动求导"><a href="#自动求导" class="headerlink" title="自动求导"></a>自动求导</h2><ol><li><p>简单情况</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">import torch</span><br><span class="line">from torch.autograd import Variable</span><br><span class="line"></span><br><span class="line">x = Variable(torch.Tensor([2]), requires_grad=True)</span><br><span class="line">y = x + 2</span><br><span class="line">z = y ** 2 + 3</span><br><span class="line"></span><br><span class="line">z.backward()</span><br><span class="line">print(x.grad)</span><br></pre></td></tr></table></figure></li><li><p>复杂情况</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">m = Variable(torch.FloatTensor([[2, 3]]), requires_grad=True) # 构建一个 1 x 2 的矩阵</span><br><span class="line">n = Variable(torch.zeros(1, 2)) # 构建一个相同大小的 0 矩阵</span><br><span class="line"></span><br><span class="line">n[0, 0] = m[0, 0] ** 2</span><br><span class="line">n[0, 1] = m[0, 1] ** 3</span><br><span class="line"></span><br><span class="line">n.backward(torch.ones_like(n)) # 将 (w0, w1) 取成 (1, 1)</span><br><span class="line">print(m.grad)</span><br></pre></td></tr></table></figure></li><li><p>多次自动求导</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">x = Variable(torch.FloatTensor([3]), requires_grad=True)</span><br><span class="line">y = x * 2 + x ** 2 + 3</span><br><span class="line"></span><br><span class="line">y.backward(retain_graph=True) # 设置 retain_graph 为 True 来保留计算图</span><br><span class="line">print(x.grad) # 8</span><br><span class="line"></span><br><span class="line">y.backward() # 再做一次自动求导，这次不保留计算图</span><br><span class="line">print(x.grad) # 16</span><br></pre></td></tr></table></figure></li></ol><p>这里做了两次自动求导，16 为第一次的梯度 8 和第二次的梯度 8 加和结果。</p><ol><li>练习<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">x = Variable(torch.FloatTensor([2, 3]), requires_grad=True)</span><br><span class="line">k = Variable(torch.zeros(2))</span><br><span class="line"></span><br><span class="line">k[0] = x[0] ** 2 + 3 * x[1]</span><br><span class="line">k[1] = x[1] ** 2 + 2 * x[0]</span><br><span class="line"></span><br><span class="line">j = torch.zeros(2, 2)</span><br><span class="line"></span><br><span class="line">k.backward(torch.FloatTensor([1, 0]), retain_graph=True)</span><br><span class="line">j[0] = x.grad.data</span><br><span class="line"></span><br><span class="line">x.grad.data.zero_() # 归零之前求得的梯度</span><br><span class="line"></span><br><span class="line">k.backward(torch.FloatTensor([0, 1]))</span><br><span class="line">j[1] = x.grad.data</span><br></pre></td></tr></table></figure></li></ol><h2 id="线性模型和梯度下降"><a href="#线性模型和梯度下降" class="headerlink" title="线性模型和梯度下降"></a>线性模型和梯度下降</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br></pre></td><td class="code"><pre><span class="line">import torch</span><br><span class="line">import numpy as np</span><br><span class="line">from torch.autograd import Variable</span><br><span class="line"></span><br><span class="line">torch.manual_seed(2017)</span><br><span class="line"></span><br><span class="line"># 读入数据 x 和 y</span><br><span class="line">x_train = np.array([[3.3], [4.4], [5.5], [6.71], [6.93], [4.168],</span><br><span class="line">                    [9.779], [6.182], [7.59], [2.167], [7.042],</span><br><span class="line">                    [10.791], [5.313], [7.997], [3.1]], dtype=np.float32)</span><br><span class="line"></span><br><span class="line">y_train = np.array([[1.7], [2.76], [2.09], [3.19], [1.694], [1.573],</span><br><span class="line">                    [3.366], [2.596], [2.53], [1.221], [2.827],</span><br><span class="line">                    [3.465], [1.65], [2.904], [1.3]], dtype=np.float32)</span><br><span class="line"></span><br><span class="line"># 画出图像</span><br><span class="line">import matplotlib.pyplot as plt</span><br><span class="line">%matplotlib inline</span><br><span class="line"></span><br><span class="line">plt.plot(x_train, y_train, &apos;bo&apos;)</span><br><span class="line"></span><br><span class="line"># 转换成 Tensor</span><br><span class="line">x_train = torch.from_numpy(x_train)</span><br><span class="line">y_train = torch.from_numpy(y_train)</span><br><span class="line"></span><br><span class="line"># 定义参数 w 和 b</span><br><span class="line">w = Variable(torch.randn(1), requires_grad=True) # 随机初始化</span><br><span class="line">b = Variable(torch.zeros(1), requires_grad=True) # 使用 0 进行初始化</span><br><span class="line"></span><br><span class="line"># 构建线性回归模型</span><br><span class="line">x_train = Variable(x_train)</span><br><span class="line">y_train = Variable(y_train)</span><br><span class="line"></span><br><span class="line">def linear_model(x):</span><br><span class="line">    return x * w + b</span><br><span class="line"></span><br><span class="line">y_ = linear_model(x_train)</span><br><span class="line"></span><br><span class="line"># 模型输出</span><br><span class="line">plt.plot(x_train.data.numpy(), y_train.data.numpy(), &apos;bo&apos;, label=&apos;real&apos;)</span><br><span class="line">plt.plot(x_train.data.numpy(), y_.data.numpy(), &apos;ro&apos;, label=&apos;estimated&apos;)</span><br><span class="line">plt.legend()</span><br><span class="line"></span><br><span class="line"># 计算误差</span><br><span class="line">def get_loss(y_, y):</span><br><span class="line">    return torch.mean((y_ - y_train) ** 2)</span><br><span class="line"></span><br><span class="line">loss = get_loss(y_, y_train)</span><br><span class="line"></span><br><span class="line"># 自动求导</span><br><span class="line">loss.backward()</span><br><span class="line"></span><br><span class="line"># 更新一次参数</span><br><span class="line">w.data = w.data - 1e-2 * w.grad.data</span><br><span class="line">b.data = b.data - 1e-2 * b.grad.data</span><br><span class="line"></span><br><span class="line"># 进行 10 次更新</span><br><span class="line">for e in range(10):</span><br><span class="line">    y_ = linear_model(x_train)</span><br><span class="line">    loss = get_loss(y_, y_train)</span><br><span class="line">    </span><br><span class="line">    w.grad.zero_() # 记得归零梯度</span><br><span class="line">    b.grad.zero_() # 记得归零梯度</span><br><span class="line">    loss.backward()</span><br><span class="line">    </span><br><span class="line">    w.data = w.data - 1e-2 * w.grad.data # 更新 w</span><br><span class="line">    b.data = b.data - 1e-2 * b.grad.data # 更新 b </span><br><span class="line">    print(&apos;epoch: &#123;&#125;, loss: &#123;&#125;&apos;.format(e, loss.data[0]))</span><br></pre></td></tr></table></figure><ol><li><p>获得 <code>[x,x^2,x^3]</code>：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">x_sample = np.arange(-3, 3.1, 0.1)</span><br><span class="line">x_train = np.stack([x_sample ** i for i in range(1, 4)], axis=1)</span><br></pre></td></tr></table></figure></li><li><p>线性模型</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">def multi_linear(x):</span><br><span class="line">    return torch.mm(x, w) + b</span><br></pre></td></tr></table></figure></li></ol><h2 id="Logistic-回归"><a href="#Logistic-回归" class="headerlink" title="Logistic 回归"></a>Logistic 回归</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br></pre></td><td class="code"><pre><span class="line">import torch</span><br><span class="line">from torch.autograd import Variable</span><br><span class="line">import numpy as np</span><br><span class="line">import matplotlib.pyplot as plt</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"># 设定随机种子</span><br><span class="line">torch.manual_seed(2017)</span><br><span class="line"></span><br><span class="line"># 从 data.txt 中读入点</span><br><span class="line">with open(&apos;./data.txt&apos;, &apos;r&apos;) as f:</span><br><span class="line">    data_list = [i.split(&apos;\n&apos;)[0].split(&apos;,&apos;) for i in f.readlines()]</span><br><span class="line">    data = [(float(i[0]), float(i[1]), float(i[2])) for i in data_list]</span><br><span class="line"></span><br><span class="line"># 标准化</span><br><span class="line">x0_max = max([i[0] for i in data])</span><br><span class="line">x1_max = max([i[1] for i in data])</span><br><span class="line">data = [(i[0]/x0_max, i[1]/x1_max, i[2]) for i in data]</span><br><span class="line"></span><br><span class="line">x0 = list(filter(lambda x: x[-1] == 0.0, data)) # 选择第一类的点</span><br><span class="line">x1 = list(filter(lambda x: x[-1] == 1.0, data)) # 选择第二类的点</span><br><span class="line"></span><br><span class="line">np_data = np.array(data, dtype=&apos;float32&apos;) # 转换成 numpy array</span><br><span class="line">x_data = torch.from_numpy(np_data[:, 0:2]) # 转换成 Tensor, 大小是 [100, 2]</span><br><span class="line">y_data = torch.from_numpy(np_data[:, -1]).unsqueeze(1) # 转换成 Tensor，大小是 [100, 1]</span><br><span class="line"></span><br><span class="line"># 定义 sigmoid 函数</span><br><span class="line">def sigmoid(x):</span><br><span class="line">    return 1 / (1 + np.exp(-x))</span><br><span class="line"></span><br><span class="line">import torch.nn.functional as F</span><br><span class="line"></span><br><span class="line"># 计算loss</span><br><span class="line">def binary_loss(y_pred, y):</span><br><span class="line">    logits = (y * y_pred.clamp(1e-12).log() + (1 - y) * (1 - y_pred).clamp(1e-12).log()).mean() # clamp 约束最大值最小值</span><br><span class="line">    return -logits</span><br><span class="line"></span><br><span class="line"># 使用 torch.optim 更新参数</span><br><span class="line">from torch import nn</span><br><span class="line">w = nn.Parameter(torch.randn(2, 1))</span><br><span class="line">b = nn.Parameter(torch.zeros(1))</span><br><span class="line"></span><br><span class="line">def logistic_regression(x):</span><br><span class="line">    return F.sigmoid(torch.mm(x, w) + b)</span><br><span class="line"></span><br><span class="line">optimizer = torch.optim.SGD([w, b], lr=1.)</span><br><span class="line"></span><br><span class="line"># 进行 1000 次更新</span><br><span class="line">import time</span><br><span class="line"></span><br><span class="line">start = time.time()</span><br><span class="line">for e in range(1000):</span><br><span class="line">    # 前向传播</span><br><span class="line">    y_pred = logistic_regression(x_data)</span><br><span class="line">    loss = binary_loss(y_pred, y_data) # 计算 loss</span><br><span class="line">    # 反向传播</span><br><span class="line">    optimizer.zero_grad() # 使用优化器将梯度归 0</span><br><span class="line">    loss.backward()</span><br><span class="line">    optimizer.step() # 使用优化器来更新参数</span><br><span class="line">    # 计算正确率</span><br><span class="line">    mask = y_pred.ge(0.5).float()</span><br><span class="line">    acc = (mask == y_data).sum().data[0] / y_data.shape[0]</span><br><span class="line">    if (e + 1) % 200 == 0:</span><br><span class="line">        print(&apos;epoch: &#123;&#125;, Loss: &#123;:.5f&#125;, Acc: &#123;:.5f&#125;&apos;.format(e+1, loss.data[0], acc))</span><br><span class="line">during = time.time() - start</span><br><span class="line">print()</span><br><span class="line">print(&apos;During Time: &#123;:.3f&#125; s&apos;.format(during))</span><br></pre></td></tr></table></figure><ul><li>pytorch 中包含一些常见 <a href="https://pytorch.org/docs/0.3.0/nn.html#loss-functions" target="_blank" rel="noopener">loss</a>，如线性回归分类<code>nn.MSE()</code>和二分类<code>nn.BCEWithLogitsLoss()</code><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><span class="line"># 使用自带的loss</span><br><span class="line">criterion = nn.BCEWithLogitsLoss() # 将 sigmoid 和 loss 写在一层，有更快的速度、更好的稳定性</span><br><span class="line"></span><br><span class="line">w = nn.Parameter(torch.randn(2, 1))</span><br><span class="line">b = nn.Parameter(torch.zeros(1))</span><br><span class="line"></span><br><span class="line">def logistic_reg(x):</span><br><span class="line">    return torch.mm(x, w) + b</span><br><span class="line"></span><br><span class="line">optimizer = torch.optim.SGD([w, b], 1.)</span><br><span class="line"></span><br><span class="line">y_pred = logistic_reg(x_data)</span><br><span class="line">loss = criterion(y_pred, y_data)</span><br><span class="line">print(loss.data)</span><br><span class="line"></span><br><span class="line"># 同样进行 1000 次更新</span><br><span class="line">start = time.time()</span><br><span class="line">for e in range(1000):</span><br><span class="line">    # 前向传播</span><br><span class="line">    y_pred = logistic_reg(x_data)</span><br><span class="line">    loss = criterion(y_pred, y_data)</span><br><span class="line">    # 反向传播</span><br><span class="line">    optimizer.zero_grad()</span><br><span class="line">    loss.backward()</span><br><span class="line">    optimizer.step()</span><br><span class="line">    # 计算正确率</span><br><span class="line">    mask = y_pred.ge(0.5).float()</span><br><span class="line">    acc = (mask == y_data).sum().data / y_data.shape[0]</span><br><span class="line">    if (e + 1) % 200 == 0:</span><br><span class="line">        print(&apos;epoch: &#123;&#125;, Loss: &#123;:.5f&#125;, Acc: &#123;:.5f&#125;&apos;.format(e+1, loss.data, acc))</span><br><span class="line"></span><br><span class="line">during = time.time() - start</span><br><span class="line">print()</span><br><span class="line">print(&apos;During Time: &#123;:.3f&#125; s&apos;.format(during))</span><br></pre></td></tr></table></figure></li></ul>]]></content>
      
      
      <categories>
          
          <category> Pytorch </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
            <tag> Pytorch </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Attention</title>
      <link href="/2019/12/04/attention/"/>
      <url>/2019/12/04/attention/</url>
      
        <content type="html"><![CDATA[<h1 id="Attention"><a href="#Attention" class="headerlink" title="Attention"></a>Attention</h1><p>参考：<a href="https://towardsdatascience.com/attn-illustrated-attention-5ec4ad276ee3" target="_blank" rel="noopener">Attn: Illustrated Attention</a></p><p>随着深度学习的发展，Neural Machine Translation(NMT) 也逐渐取代传统的 Statistical Machine Translation(SMT)。 其中，最广为人知的框架为 <a href="https://arxiv.org/abs/1409.3215" target="_blank" rel="noopener">Sutskever et. al</a> 提出的 sequence-to-sequence(seq2seq) 模型。</p><p>Seq2seq 是一个含有两个 RNN 的 encoder-decoder 模型： encoder，按字符读入从而获得一个固定长度的表示； decoder,根据这些输入训练另一个 RNN 从而获得顺序输出。</p><p>这样的模型会导致最终 decoder 获得的是 encoder 最后输出的 hidden state，当文本过长时，容易遗忘开始输入的字段。</p><p>Attention 可以作为 encoder 与 decoder 的中间接口，为 decoder 提供每一个 encoder hidden state 的信息。如此，模型能够选择性的关注有用的部分，并学习 encoder 与 decoder 中字句的对齐。</p><p>Attention 有两种类型： global attention 使用所有的 encoder hidden state，local attention 只使用部分。 Attention layer 的实现可分为4个步骤。</p><ul><li><p>准备 hidden states.<br>例子中包含4个 encoder hidden states 和 current decoder hidden state.<br>Note: 最后一个 encoder hidden state 作为 decoder 的第一个 time step 输入。 第一个 time step 的输出被称为第一个 decoder hidden state.</p></li><li><p>计算每个 encoder hidden state 的 score.<br>可以通过一个 score function（也被称为 alignment score function 或 alignment model）来计算 score。在本例中，score function 为 dot product。更多 <a href="https://towardsdatascience.com/attn-illustrated-attention-5ec4ad276ee3#ba24" target="_blank" rel="noopener">score function</a>。<br>计算公式：<code>score = decoder hidden state x encoder hidden state</code></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">decoder_hidden = [10, 5, 10]</span><br><span class="line">encoder_hidden  score</span><br><span class="line">---------------------</span><br><span class="line">     [0, 1, 1]     15 (= 10×0 + 5×1 + 10×1, the dot product)</span><br><span class="line">     [5, 0, 1]     60</span><br><span class="line">     [1, 1, 0]     15</span><br><span class="line">     [0, 5, 1]     35</span><br></pre></td></tr></table></figure></li><li><p>将所有 scores 通过一个 softmax 层。<br>通过 softmax 层，将所有 softmaxed scores 加和为1，被称为 attention distribution.</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">encoder_hidden  score  score^</span><br><span class="line">-----------------------------</span><br><span class="line">     [0, 1, 1]     15       0</span><br><span class="line">     [5, 0, 1]     60       1</span><br><span class="line">     [1, 1, 0]     15       0</span><br><span class="line">     [0, 5, 1]     35       0</span><br></pre></td></tr></table></figure></li><li><p>将每个 encoder hidden state 乘以对应 softmaxed score.<br>通过 encoder hidden state 乘以相应 softmaxed score，可以获得 alignment vector 或 annotation vector。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">encoder  score  score^  alignment</span><br><span class="line">---------------------------------</span><br><span class="line">[0, 1, 1]   15      0   [0, 0, 0]</span><br><span class="line">[5, 0, 1]   60      1   [5, 0, 1]</span><br><span class="line">[1, 1, 0]   15      0   [0, 0, 0]</span><br><span class="line">[0, 5, 1]   35      0   [0, 0, 0]</span><br></pre></td></tr></table></figure></li></ul><p>这里，表示第一个翻译的词与嵌入<code>[5,0,1]</code>的输入相对应。</p><ul><li><p>将 alignment vectors 相加。<br>将 alignment vectors 相加获得 context vector。</p></li><li><p>将 context vector 输入 decoder.</p></li></ul><h1 id="Self-attention"><a href="#Self-attention" class="headerlink" title="Self-attention"></a>Self-attention</h1><p>参考：<a href="https://towardsdatascience.com/illustrated-self-attention-2d627e33b20a" target="_blank" rel="noopener">Illustrated: Self-Attention</a></p><ul><li><p>输入<br>例子中，输入为3个4维向量：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">Input 1: [1, 0, 1, 0] </span><br><span class="line">Input 2: [0, 2, 0, 2]</span><br><span class="line">Input 3: [1, 1, 1, 1]</span><br></pre></td></tr></table></figure></li><li><p>初始化权值<br>每个输入有三个表示：key, query, value。例子中，这些表示用3维向量表示，则权值为4*3矩阵。<br>Note: value的维度与输出相同。</p><ul><li><p>key 的权值</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[[0, 0, 1],</span><br><span class="line"> [1, 1, 0],</span><br><span class="line"> [0, 1, 0],</span><br><span class="line"> [1, 1, 0]]</span><br></pre></td></tr></table></figure></li><li><p>query 的权值</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[[1, 0, 1],</span><br><span class="line"> [1, 0, 0],</span><br><span class="line"> [0, 0, 1],</span><br><span class="line"> [0, 1, 1]]</span><br></pre></td></tr></table></figure></li><li><p>value 的权值</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[[0, 2, 0],</span><br><span class="line"> [0, 3, 0],</span><br><span class="line"> [1, 0, 3],</span><br><span class="line"> [1, 1, 0]]</span><br></pre></td></tr></table></figure></li></ul><p>Nots: 在神经网络中，权值常由合适的随机分布来初始化，如 Gaussian, Xavier 和 Kaiming 分布。</p></li><li><p>计算 key, query 和 value<br>计算公式为：<code>input x weight</code></p><ul><li><p>key:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">               [0, 0, 1]</span><br><span class="line">[1, 0, 1, 0]   [1, 1, 0]   [0, 1, 1]</span><br><span class="line">[0, 2, 0, 2] x [0, 1, 0] = [4, 4, 0]</span><br><span class="line">[1, 1, 1, 1]   [1, 1, 0]   [2, 3, 1]</span><br></pre></td></tr></table></figure></li><li><p>query:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">               [1, 0, 1]</span><br><span class="line">[1, 0, 1, 0]   [1, 0, 0]   [1, 0, 2]</span><br><span class="line">[0, 2, 0, 2] x [0, 0, 1] = [2, 2, 2]</span><br><span class="line">[1, 1, 1, 1]   [0, 1, 1]   [2, 1, 3]</span><br></pre></td></tr></table></figure></li><li><p>value:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">               [0, 2, 0]</span><br><span class="line">[1, 0, 1, 0]   [0, 3, 0]   [1, 2, 3] </span><br><span class="line">[0, 2, 0, 2] x [1, 0, 3] = [2, 8, 0]</span><br><span class="line">[1, 1, 1, 1]   [1, 1, 0]   [2, 6, 3]</span><br></pre></td></tr></table></figure></li></ul><p>Note: 有时候也可以加上偏置。</p></li><li><p>计算 Input 1 的 attention score<br>计算公式为：<code>Input 1&#39;s query x keys^T</code></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">            [0, 4, 2]</span><br><span class="line">[1, 0, 2] x [1, 4, 3] = [2, 4, 4]</span><br><span class="line">            [1, 0, 1]</span><br></pre></td></tr></table></figure></li></ul><p>Note: 上述为 dot product attention, 其他 <a href="https://towardsdatascience.com/attn-illustrated-attention-5ec4ad276ee3" target="_blank" rel="noopener">score function</a> 有 scaled dot product 和 additive/concat。</p><ul><li><p>计算softmax<br>对 attention score 进行 softmax 计算:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">softmax([2, 4, 4]) = [0.0, 0.5, 0.5]</span><br></pre></td></tr></table></figure></li><li><p>将 score 乘以 value:<br>将 attention score 乘以对应的 value, 获得 weighted values:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">1: 0.0 * [1, 2, 3] = [0.0, 0.0, 0.0]</span><br><span class="line">2: 0.5 * [2, 8, 0] = [1.0, 4.0, 0.0]</span><br><span class="line">3: 0.5 * [2, 6, 3] = [1.0, 3.0, 1.5]</span><br></pre></td></tr></table></figure></li><li><p>将 weighted values 加和获得 Output 1:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">  [0.0, 0.0, 0.0]</span><br><span class="line">+ [1.0, 4.0, 0.0]</span><br><span class="line">+ [1.0, 3.0, 1.5]</span><br><span class="line">-----------------</span><br><span class="line">= [2.0, 7.0, 1.5]</span><br></pre></td></tr></table></figure></li><li><p>对于 Input 2 和 Input 3 重复4-7操作<br>Note: query 和 key 的维度需要保持一致，而 value 的维度与 output 一致。</p></li></ul>]]></content>
      
      
      <categories>
          
          <category> Machine Learning </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Jupyter使用笔记</title>
      <link href="/2019/11/12/jupyter/"/>
      <url>/2019/11/12/jupyter/</url>
      
        <content type="html"><![CDATA[<p>参考：《<a href="https://zhuanlan.zhihu.com/p/33105153" target="_blank" rel="noopener">Jupyter Notebook介绍、安装及使用教程</a>》， 《<a href="https://blog.csdn.net/wangdan113269/article/details/88994792" target="_blank" rel="noopener">远程连接服务器</a>》</p><ul><li><p>修改密码：</p>  <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ jupyter notebook password</span><br></pre></td></tr></table></figure></li><li><p>生成密钥（较为复杂）：</p>  <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ ipython</span><br><span class="line"></span><br><span class="line">from notebook.auth import passwd</span><br><span class="line">passwd()</span><br></pre></td></tr></table></figure><p>  输入密码后生成密钥，复制该密钥。</p></li><li><p>生成配置文件</p>  <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ jupyter notebook --generate-config</span><br></pre></td></tr></table></figure></li><li><p>更改配置文件</p>  <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">$vim ~/.jupyter/jupyter_notebook_config.py</span><br><span class="line"></span><br><span class="line">c.NotebookApp.ip=&apos;*&apos;</span><br><span class="line">c.NotebookApp.password = u&apos;粘贴密钥&apos;</span><br><span class="line">c.NotebookApp.open_browser = False</span><br><span class="line">c.NotebookApp.port =8888</span><br></pre></td></tr></table></figure><p>  端口号可根据自己服务器修改。保存后退出。</p></li><li><p>启动</p>  <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$jupyter notebook --allow-root</span><br></pre></td></tr></table></figure><p>  在浏览器中打开</p>  <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">http://服务器IP:端口号</span><br></pre></td></tr></table></figure></li><li><p>修改主页面</p>  <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$vim ~/.jupyter/jupyter_notebook_config.py</span><br><span class="line"></span><br><span class="line">c.NotebookApp.notebook_dir = &apos;主页面地址&apos;</span><br></pre></td></tr></table></figure></li><li><p>显示所有数据<br>  <code>np.set_printoptions(threshold=np.inf)</code></p></li><li>文件中的python代码有大量输出时会导致notebook文件特别大以至于打不开。解决思路是清除文件中的output内容。<br>  <code>jupyter nbconvert --ClearOutputPreprocessor.enabled=True --to notebook --output=newipy .\filename.ipynb</code><br>  命令参数解读<br>  <code>.\filename.ipynb</code> ：是你打不开的那个大文件<br>  <code>--output=newipy</code> ：是清除output后的ipynb文件名</li><li><p><a href="https://zhuanlan.zhihu.com/p/370024835" target="_blank" rel="noopener">切换不同内核</a></p><ul><li><p>在环境中安装ipykernel</p>  <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">conda install ipykernel</span><br></pre></td></tr></table></figure></li><li><p>将环境写入kenel中</p>  <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">python -m ipykernel install --user  --name tfgpu2.4 --display-name &quot;tensorflow-gpu2.4&quot;</span><br></pre></td></tr></table></figure></li><li><p>在base环境中启动jupyter notebook</p></li><li>新建文件的时候可直接选择不同版本的环境</li><li>打开文件后，在Jupyter Notebook界面 <code>kernel----&gt;change kernel</code> 中选择环境</li><li><p>查看环境内核</p>  <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">jupyter kernelspec list</span><br></pre></td></tr></table></figure></li><li><p>删除环境内核</p>  <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">jupyter kernelspec remove &lt;env-name&gt;</span><br></pre></td></tr></table></figure></li></ul></li></ul>]]></content>
      
      
      <categories>
          
          <category> Notes </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Jupyter </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Quality assessment</title>
      <link href="/2019/10/13/quality/"/>
      <url>/2019/10/13/quality/</url>
      
        <content type="html"><![CDATA[<h1 id="Image-Quality-Assessment-IQA"><a href="#Image-Quality-Assessment-IQA" class="headerlink" title="Image Quality Assessment(IQA)"></a>Image Quality Assessment(IQA)</h1><ol><li>有参考图图像的质量评估</li></ol><ul><li><p>MSE: 计算图像的像素差的平方，然后在全图上求平均：</p><script type="math/tex; mode=display">MSE=\frac{1}{mn}\sum_{i=0}^{m-1}\sum_{j=0}{n-1}[I(i,j) - K(i,j)]^2</script><p>但是，基于 MSE 的损失不足以表达人的视觉系统对图片的直观感受。</p></li><li><p>PSNR: 峰值信噪比(Peak-signal to Noise Ratio)</p><script type="math/tex; mode=display">PSNR=10 \times \log_{10}\frac{MAXI^2}{MSE}</script><p>MAXI表示颜色的最大数值。</p></li><li><p><a href="https://zhuanlan.zhihu.com/p/67199699" target="_blank" rel="noopener">SSIM(Structural Similarity)</a>: 根据图像的亮度(luminance)，对比度(contrast)和结构(structure)进行相似度比较：</p><ul><li><p>亮度：</p><script type="math/tex; mode=display">{\mu_x} = \frac{1}{N}\sum_{i=1}^{N}x_i</script><script type="math/tex; mode=display">l(x,y) = \frac{2\mu_x\mu_y+C_1}{\mu_x^2+\mu_y^2+C_1}</script><script type="math/tex; mode=display">C_1 = (K_1L)^2</script><p>其中K&lt;&lt;1是一个常数，常取值为0.01，L是灰度图的动态范围，由图像的数据类型决定，如果数据为 uint8 型，则 L=255。可以看出，l(x,y)对称且始终小于等于1，当 x = y 时为1。</p></li><li><p>对比度：</p><script type="math/tex; mode=display">\sigma_x = (\frac{1}{N-1}\sum_{i=1}^{N}(x_i-\mu_x)^2)^{\frac{1}{2}}</script><script type="math/tex; mode=display">c(x,y) = \frac{2\sigma_x\sigma_y+C_2}{\sigma_x^2+\sigma_y^2+C_2}</script><script type="math/tex; mode=display">C_2 = (k_2L)^2</script><p>K_2 常取值为0.03，c(x,y)对称且小于等于1，当x=y时等号成立。</p></li><li><p>结构相似度： 归一化的两个向量的相似度比较 </p></li></ul></li><li><p>基于信息论基础：信息保真度准则（Information Fidelity Criterion,IFC）和视觉信息保真度（Visual Information Fidelity,VIF）</p></li></ul><ol><li>无参考图图像的质量评估</li></ol><ul><li><p><a href="https://zhuanlan.zhihu.com/p/40746930" target="_blank" rel="noopener">UCIQE</a>: 色彩浓度、饱和度、对比度的线性组合：</p><script type="math/tex; mode=display">UCIQE = c_1\*\sigma_c+c_x\*con_1+c_3\*\mu_s</script></li><li><p>UIQM (underater image quality measurement)</p></li><li><p><a href="https://baike.baidu.com/item/IQA/19453034" target="_blank" rel="noopener">图像统计特性</a></p><ul><li>均值： 图像像素的平均值，与亮度算法一致</li><li>标准差：</li></ul></li></ul><h1 id="Video-Quality-Assessment-VQA"><a href="#Video-Quality-Assessment-VQA" class="headerlink" title="Video Quality Assessment(VQA)"></a>Video Quality Assessment(VQA)</h1><p>参考：<a href="https://blog.csdn.net/leixiaohua1020/article/details/16359465" target="_blank" rel="noopener">视频质量评价</a>, <a href="https://github.com/Netflix/vmaf">VMAF</a></p><ol><li><p>视频编码器</p></li><li><p>编码标准</p></li><li><p>视频质量评价</p></li></ol><ul><li>视频主观质量评价(Subjective Quality Assessment, SQA)<br>  DSIS, DSCQS, SSM</li><li>视频客观质量评价(Objective Quality Assessment, OQA)<br>  全参考(FullReference，FR)，部分参考(ReducedReference，RR)和无参考(No Reference，NR)<br>扩展：<a href="https://blog.csdn.net/leixiaohua1020/article/details/50534150#comments" target="_blank" rel="noopener">视音频数据处理</a></li></ul><ol><li><a href="https://www.jianshu.com/p/b97e4d15a400" target="_blank" rel="noopener">Nefix对视频源特性的分析</a>：</li></ol><ul><li>压缩失真，画质损失，随机噪声，几何形变</li><li>视频内容</li><li>源素材特征，如：胶片颗粒、传感器噪声、计算机生成的材质、亮度、对比度、颜色变化、色泽浓郁度、锐度。</li><li><a href="https://www.infoq.cn/article/a-quality-assessment-tool-for-video-streaming-media" target="_blank" rel="noopener">VMAF</a>: 视觉信息保真度(VIF)，细节丢失指标(DLM)，运动</li></ul><ol><li><a href="https://www.infoq.cn/article/GfEC9QRjRgofdA7sR_H2" target="_blank" rel="noopener">爱奇艺短视频质量评估模型</a></li></ol><ul><li>封面图质量：模糊，黑边，拉伸变形，画面暗，无主体，无意义等。</li><li>视频内容质量：视频无意义，无聊，不清晰，花屏，广告，低俗等。</li><li><p>文本质量：标题过于简单，特殊符号多，句子不通顺，语法结构不正常，标题党，图文不符等。</p></li><li><p>封面图质量模型：基于卷积模型提取的深度特征和人工设计特征的图像质量模型。</p><ul><li>基础质量特征<ul><li>边缘的空间分布：将图像进行拉普拉斯滤波与其类别拉普拉斯图像均值的 L1 距离进行度量。</li><li>颜色分布、色调计数、对比度与亮度：基于图像的RGB或HSV颜色空间来统计。</li><li>模糊程度：模糊核算法用以评估图像或图像像素的锐度或聚焦程度。6组模糊特征，及其统计均值、方差、最大值、最小值，考虑到局部模糊性，划分图像的四个区域。<ul><li>基于梯度 (Gradient-based operators)，该算法假设清晰图像相比模糊图像有更锐利的线条；</li><li>基于拉普拉斯变换 (Laplacian-based operators)，统计图像中线条的占比；</li><li>其他包括基于小波算子 (Wavelet-based operator)、基于统计算子 (Statistic-based operators)、基于离散余弦算子 (Discrete cosine transform)、基于局部表示和滤波相结合 (Miscellaneous operators)。</li></ul></li></ul></li><li>Deep &amp; Wide： Google NIMA深度美感模型</li></ul></li><li>视频内容质量模型：端到端训练的基于多模态的深度内容质量模型。<ul><li>视频抽帧表示、光流表示和音频表示</li></ul></li><li><p>文本质量模型：基于文本结构特征和文本语义特征的文本质量分类模型。</p><ul><li>语义抽取和句法结构抽取</li><li>词性、依存关系、通顺度、长度、异常字符占比、类型标签</li></ul></li><li><p>NetVlad：视觉表示</p></li><li>TSN：运动表示</li><li>VGGISH: 音频特征</li><li>XGBOOST：文本特征</li></ul><ol><li><p>MOVIE（MOtion-based Video IntegrityEvalution）：计算视频中物体的运动矢量，联合时域和空域的失真信息，最终得到一个符合主观感受的失真评价分数。</p></li><li><p>RankIQA: <a href="https://xialeiliu.github.io/RankIQA/" target="_blank" rel="noopener">gitbug</a>, <a href="https://zhuanlan.zhihu.com/p/32553977" target="_blank" rel="noopener">综述</a></p></li><li><p>VMAF: <a href="https://github.com/Netflix/vmaf">github</a>, <a href="https://www.infoq.cn/article/a-quality-assessment-tool-for-video-streaming-media" target="_blank" rel="noopener">介绍</a>, <a href="https://www.jianshu.com/p/b97e4d15a400" target="_blank" rel="noopener">使用</a></p></li><li><p>GMAD: <a href="https://ece.uwaterloo.ca/~k29ma/websites/gMAD/" target="_blank" rel="noopener">page</a></p></li><li><p>基于opencv的质量评估: <a href="https://github.com/Leezhen2014/python--/blob/master/BlurDetection.py">github</a>, <a href="https://cloud.tencent.com/developer/article/1395727" target="_blank" rel="noopener">介绍</a></p></li></ol>]]></content>
      
      
      <categories>
          
          <category> Notes </category>
          
      </categories>
      
      
        <tags>
            
            <tag> quality </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Tensorflow 小记</title>
      <link href="/2019/03/25/tensorflow3/"/>
      <url>/2019/03/25/tensorflow3/</url>
      
        <content type="html"><![CDATA[<ol><li><p>打印tensor值</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">import tensorflow as tf</span><br><span class="line">x = tf.constant(1)</span><br><span class="line">with tf.Session() as sess:</span><br><span class="line">    print sess.run(x)</span><br></pre></td></tr></table></figure></li><li><p><a href="https://blog.csdn.net/qq_16137569/article/details/72568793" target="_blank" rel="noopener">矩阵加和</a><br><code>tf.reduce_sum(input_tenosr, reduction_indices, keep_dims, name)</code></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">x = [[1,1,1], [1,1,1]]</span><br><span class="line"></span><br><span class="line">tf.reduce_sum(x) # 6</span><br><span class="line">tf.reduce_sum(x,0) # [2,2,2]</span><br><span class="line">tf.reduce_sum(x,1) # [3,3]</span><br><span class="line">tf.reduce_sum(x,1, keep_dims=True) # [[3],[3]]</span><br><span class="line">tf.reduce_sum(x,[0,1]) # 6</span><br></pre></td></tr></table></figure></li><li><p><a href="https://www.cnblogs.com/as3asddd/p/10129241.html" target="_blank" rel="noopener">获取中间某一层的输出</a></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line">from keras.models import Model</span><br><span class="line"></span><br><span class="line">model = Sequential()</span><br><span class="line">model.add(Dense(32, activation=&apos;relu&apos;, input_dim=100))</span><br><span class="line">model.add(Dense(16, activation=&apos;relu&apos;,name=&quot;Dense_1&quot;))</span><br><span class="line">model.add(Dense(1, activation=&apos;sigmoid&apos;,name=&quot;Dense_2&quot;))</span><br><span class="line"></span><br><span class="line">model.compile(optimizer=&apos;rmsprop&apos;,</span><br><span class="line">            loss=&apos;binary_crossentropy&apos;,</span><br><span class="line">            metrics=[&apos;accuracy&apos;])</span><br><span class="line"></span><br><span class="line">model.fit(data, labels, epochs=10, batch_size=32)</span><br><span class="line"></span><br><span class="line"># 已有的model在load权重过后</span><br><span class="line"># 取某一层的输出为输出新建为model，采用函数模型</span><br><span class="line">dense1_layer_model = Model(inputs=model.input,</span><br><span class="line">outputs=model.get_layer(&apos;Dense_1&apos;).output)</span><br><span class="line"></span><br><span class="line"># layer_model = Model(inputs=model.input, outputs=model.layers[6].output)</span><br><span class="line"></span><br><span class="line">#以这个model的预测值作为输出</span><br><span class="line">dense1_output = dense1_layer_model.predict(data)</span><br></pre></td></tr></table></figure></li><li><p><a href="https://blog.csdn.net/ACM_hades/article/details/88790013" target="_blank" rel="noopener">矩阵上下三角</a></p><ul><li><code>tf.matrix_band_part</code></li><li><p>2.0版本： <code>tf.linalg.band_par</code></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">tf.linalg.band_part(</span><br><span class="line">    input,</span><br><span class="line">    num_lower,</span><br><span class="line">    num_upper,</span><br><span class="line">    name=None</span><br><span class="line">)</span><br></pre></td></tr></table></figure><p>作用：主要功能是以对角线为中心，取它的副对角线部分，其他部分用0填充。<br><code>input</code>:输入的张量.<br><code>num_lower</code>:下三角矩阵保留的副对角线数量，从主对角线开始计算，相当于下三角的带宽。取值为负数时，则全部保留。<br><code>num_upper</code>:上三角矩阵保留的副对角线数量，从主对角线开始计算，相当于上三角的带宽。取值为负数时，则全部保留。</p></li></ul></li><li><p>张量扩充<br><code>tf.tile(raw, multiples=[2, 1])</code></p></li><li><p>查看gpu<br><code>tf.test.is_gpu_available()</code></p></li></ol>]]></content>
      
      
      <categories>
          
          <category> Tensorflow </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
            <tag> Tensorflow </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Pandas学习笔记</title>
      <link href="/2019/03/22/pandas/"/>
      <url>/2019/03/22/pandas/</url>
      
        <content type="html"><![CDATA[<p>记录一些pandas的语法<br><a href="http://pandas.pydata.org/pandas-docs/stable/getting_started/comparison/comparison_with_sql.html" target="_blank" rel="noopener">官方文档</a>, <a href="http://pandas.pydata.org/pandas-docs/stable/getting_started/comparison/comparison_with_sql.html" target="_blank" rel="noopener">pandas vs. SQL</a>, <a href="https://www.cnblogs.com/en-heng/p/5630849.html" target="_blank" rel="noopener">教程</a></p><ol><li><p>读取与存储<br><code>read_csv</code>, <code>to_csv</code>, <code>read_pickle</code>,<code>to_pickle</code><br>当某列值为list的时候，采用<code>to_pickle</code>存储，如果用<code>to_csv</code>存储，list会被存为str。</p></li><li><p>缺损值填充</p><ul><li><code>data = data.fillna(&#39;-1&#39;, )</code></li><li><code>data = data.fillna(data.mean(), )</code></li><li>填充<code>[]</code>: 不过，尽量不要用DataFrame存储list<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">nan_index = data[data.isnull()].index</span><br><span class="line">data.loc[nan_index,] = [[]]</span><br></pre></td></tr></table></figure></li></ul></li><li><p>计算缺损值个数：<br><code>Nan_num = data.shape[1]-data.count(axis=1)</code></p></li><li><p>去掉缺损值过多的行：<br><code>data.drop(Nan_num[Nan_num&gt;256.index.tolist(),inplace=True)</code><br>drop函数默认删除行，列需要加<code>axis = 1</code><br><code>df.drop(&#39;column_name&#39;, axis=1, inplace=True)</code><br><code>items_one = items.drop_duplicates(&#39;ITEMID&#39;,&#39;first&#39;,inplace=False)</code></p></li><li><p>df.info()查看数据type</p></li><li><p><a href="https://juejin.im/post/5acc36e66fb9a028d043c2a5" target="_blank" rel="noopener">数据转换</a><br><code>astype()</code>强制转换，仅返回数据的副本而不原地修改。</p><ul><li><p>自定义转换：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">def convert_currency(val):</span><br><span class="line">    &quot;&quot;&quot;</span><br><span class="line">    Convert the string number value to a float</span><br><span class="line">     - Remove $</span><br><span class="line">     - Remove commas</span><br><span class="line">     - Convert to float type</span><br><span class="line">    &quot;&quot;&quot;</span><br><span class="line">    new_val = val.replace(&apos;,&apos;,&apos;&apos;).replace(&apos;$&apos;, &apos;&apos;)</span><br><span class="line">    return float(new_val)</span><br><span class="line"></span><br><span class="line">df[&apos;2016&apos;].apply(convert_currency)</span><br></pre></td></tr></table></figure></li><li><p>使用lambda进行转换<br><code>df[&#39;2016&#39;].apply(lambda x: x.replace(&#39;$&#39;, &#39;&#39;).replace(&#39;,&#39;, &#39;&#39;)).astype(&#39;float&#39;)</code></p></li><li><p>使用to_numeric进行转换<br><code>pd.to_numeric(df[&#39;Jan Units&#39;], errors=&#39;coerce&#39;).fillna(0)</code></p></li></ul></li><li><p><a href="https://morvanzhou.github.io/tutorials/data-manipulation/np-pd/3-2-pd-indexing/" target="_blank" rel="noopener">选择数据</a></p><ul><li><code>loc</code>根据标签</li><li><code>iloc</code>根据序列</li><li><code>ix</code>混合</li></ul></li><li><p><a href="https://blog.csdn.net/stevenkwong/article/details/52528616" target="_blank" rel="noopener">数据合并</a></p></li><li><p><a href="https://blog.csdn.net/waple_0820/article/details/80514073" target="_blank" rel="noopener">统计个数</a></p><ul><li><code>df.groupby([&#39;id&#39;],as_index=False)[&#39;id&#39;].agg({&#39;cnt&#39;:&#39;count&#39;})</code></li><li><code>df[&#39;id&#39;].value_counts()</code></li></ul></li><li><p>datetime</p><ul><li><code>pd.to_datetime(df)</code>：将str转换为datetime</li><li><code>df.dt.year</code>: 获得datetime数据中的year</li><li><code>df.map(lambda x:x.strftime(&#39;%Y&#39;))</code></li></ul></li><li><p><a href="https://blog.csdn.net/xiaodongxiexie/article/details/53108959" target="_blank" rel="noopener">对行和列的操作</a></p></li><li><p><a href="https://amberwest.github.io/2019/03/05/%E5%88%A0%E9%99%A4DataFrame%E4%B8%AD%E6%9F%90%E5%88%97%E5%80%BC%E4%B8%BANaN%E7%9A%84%E8%AE%B0%E5%BD%95-%E8%A1%8C/" target="_blank" rel="noopener">去掉含nan的行/列</a><br><code>dropna()</code></p></li><li><p>删去重复列<br><code>DataFrame.drop_duplicates(subset=None, keep=&#39;first&#39;, inplace=False)</code></p><ul><li><code>subset</code>: 指定特定的列，默认全部</li><li><code>keep: {‘first’, ‘last’, False}, default ‘first’</code>: 删除重复项，并保留，默认第一次出现的</li></ul></li><li><p>删除某值<br><code>data2 = data[~data.isin([&#39;\\N&#39;])]</code></p></li></ol>]]></content>
      
      
      <categories>
          
          <category> Python </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Python </tag>
            
            <tag> Pandas </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Tensorflow 课堂笔记</title>
      <link href="/2019/03/09/tensorflow/"/>
      <url>/2019/03/09/tensorflow/</url>
      
        <content type="html"><![CDATA[<p>最近COURSERA上面开了<a href="https://www.coursera.org/learn/introduction-tensorflow/home/welcome" target="_blank" rel="noopener">Tensorflow的课程</a>，便去学习了一下。Github上面也放了<a href="https://github.com/lmoroney/dlaicourse">课堂笔记</a></p><h2 id="Week-1：简单网络构建"><a href="#Week-1：简单网络构建" class="headerlink" title="Week 1：简单网络构建"></a>Week 1：简单网络构建</h2><ul><li><p>Import</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">import tensorflow as tf</span><br><span class="line">import numpy as np</span><br><span class="line">from tensorflow import keras</span><br></pre></td></tr></table></figure></li><li><p>构建网络</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">model = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])])</span><br><span class="line">model.compile(optimizer=&apos;sgd&apos;, loss=&apos;mean_squared_error&apos;)</span><br></pre></td></tr></table></figure><p>构建了一个神经网络，含有一层隐层，一个神经元，输入为一维。<br>采用<code>sgd</code>优化及<code>mean_squared_error</code>损失。</p></li><li><p><a href="https://morvanzhou.github.io/tutorials/machine-learning/ML-intro/3-06-speed-up-learning/" target="_blank" rel="noopener">其他优化方法</a>: Momentum, AdaGrad, RMSProp, Adam</p></li><li><p><a href="https://keras.io/zh/losses/" target="_blank" rel="noopener">损失函数</a></p></li><li><p>输入数据</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">xs = np.array([-1.0,  0.0, 1.0, 2.0, 3.0, 4.0], dtype=float)</span><br><span class="line">ys = np.array([-3.0, -1.0, 1.0, 3.0, 5.0, 7.0], dtype=float)</span><br></pre></td></tr></table></figure></li><li><p>训练网络</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">model.fit(xs, ys, epochs=500)</span><br><span class="line">print(model.predict([10.0]))</span><br></pre></td></tr></table></figure></li></ul><h2 id="Week-2-图像分类"><a href="#Week-2-图像分类" class="headerlink" title="Week 2: 图像分类"></a>Week 2: 图像分类</h2><p>介绍Fashion AI以及一个可用数据集：<a href="https://github.com/zalandoresearch/fashion-mnist">Fashion-mnist</a><br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><span class="line">import tensorflow as tf</span><br><span class="line">print(tf.__version__)</span><br><span class="line"></span><br><span class="line">mnist = tf.keras.datasets.mnist</span><br><span class="line"></span><br><span class="line">(training_images, training_labels) ,  (test_images, test_labels) = mnist.load_data()</span><br><span class="line"></span><br><span class="line">training_images = training_images/255.0</span><br><span class="line">test_images = test_images/255.0</span><br><span class="line"></span><br><span class="line">model = tf.keras.models.Sequential([tf.keras.layers.Flatten(),</span><br><span class="line">                                    tf.keras.layers.Dense(1024, activation=tf.nn.relu),</span><br><span class="line">                                    tf.keras.layers.Dense(10, activation=tf.nn.softmax)])</span><br><span class="line"></span><br><span class="line">model.compile(optimizer = &apos;adam&apos;,</span><br><span class="line">              loss = &apos;sparse_categorical_crossentropy&apos;)</span><br><span class="line"></span><br><span class="line"># model.compile(optimizer = tf.train.AdamOptimizer(),</span><br><span class="line">              loss = &apos;sparse_categorical_crossentropy&apos;,</span><br><span class="line">              metrics=[&apos;accuracy&apos;])</span><br><span class="line"></span><br><span class="line">model.fit(training_images, training_labels, epochs=5)</span><br><span class="line"></span><br><span class="line">model.evaluate(test_images, test_labels)</span><br><span class="line"></span><br><span class="line">classifications = model.predict(test_images)</span><br><span class="line"></span><br><span class="line">print(classifications[0])</span><br><span class="line">print(test_labels[0])</span><br></pre></td></tr></table></figure></p><p>添加 Callback:<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line">import tensorflow as tf</span><br><span class="line">print(tf.__version__)</span><br><span class="line"></span><br><span class="line"># 当loss小于0.4时，结束训练</span><br><span class="line">class myCallback(tf.keras.callbacks.Callback):</span><br><span class="line">  def on_epoch_end(self, epoch, logs=&#123;&#125;):</span><br><span class="line">    if(logs.get(&apos;loss&apos;)&lt;0.4):</span><br><span class="line">      print(&quot;\nReached 60% accuracy so cancelling training!&quot;)</span><br><span class="line">      self.model.stop_training = True</span><br><span class="line"></span><br><span class="line">callbacks = myCallback()</span><br><span class="line">mnist = tf.keras.datasets.fashion_mnist</span><br><span class="line">(training_images, training_labels), (test_images, test_labels) = mnist.load_data()</span><br><span class="line">training_images=training_images/255.0</span><br><span class="line">test_images=test_images/255.0</span><br><span class="line">model = tf.keras.models.Sequential([</span><br><span class="line">  tf.keras.layers.Flatten(),</span><br><span class="line">  tf.keras.layers.Dense(512, activation=tf.nn.relu),</span><br><span class="line">  tf.keras.layers.Dense(10, activation=tf.nn.softmax)</span><br><span class="line">])</span><br><span class="line">model.compile(optimizer=&apos;adam&apos;, loss=&apos;sparse_categorical_crossentropy&apos;)</span><br><span class="line">model.fit(training_images, training_labels, epochs=5, callbacks=[callbacks])</span><br></pre></td></tr></table></figure></p><h2 id="Week-3-CNN"><a href="#Week-3-CNN" class="headerlink" title="Week 3: CNN"></a>Week 3: CNN</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line">import tensorflow as tf</span><br><span class="line">print(tf.__version__)</span><br><span class="line"></span><br><span class="line">mnist = tf.keras.datasets.fashion_mnist</span><br><span class="line">(training_images, training_labels), (test_images, test_labels) = mnist.load_data()</span><br><span class="line">training_images = training_images.reshape(60000, 28, 28, 1)</span><br><span class="line">training_images = training_images / 255.0</span><br><span class="line">test_images = test_images.reshape(10000, 28, 28, 1)</span><br><span class="line">test_images = test_images/255.0</span><br><span class="line"></span><br><span class="line">model = tf.keras.models.Sequential([</span><br><span class="line">  tf.keras.layers.Conv2D(64, (3,3), activation=&apos;relu&apos;, input_shape=(28, 28, 1)),</span><br><span class="line">  tf.keras.layers.MaxPooling2D(2, 2),</span><br><span class="line">  tf.keras.layers.Conv2D(64, (3,3), activation=&apos;relu&apos;),</span><br><span class="line">  tf.keras.layers.MaxPooling2D(2,2),</span><br><span class="line">  tf.keras.layers.Flatten(),</span><br><span class="line">  tf.keras.layers.Dense(128, activation=&apos;relu&apos;),</span><br><span class="line">  tf.keras.layers.Dense(10, activation=&apos;softmax&apos;)</span><br><span class="line">])</span><br><span class="line">model.compile(optimizer=&apos;adam&apos;, loss=&apos;sparse_categorical_crossentropy&apos;, metrics=[&apos;accuracy&apos;])</span><br><span class="line">model.summary()</span><br><span class="line">model.fit(training_images, training_labels, epochs=5)</span><br><span class="line">test_loss = model.evaluate(test_images, test_labels)</span><br></pre></td></tr></table></figure><ul><li><code>model.summary()</code> 绘制网络结构</li></ul><p>图像展示:<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line">import matplotlib.pyplot as plt</span><br><span class="line"></span><br><span class="line">f, axarr = plt.subplots(3,4)</span><br><span class="line">FIRST_IMAGE=0</span><br><span class="line">SECOND_IMAGE=7</span><br><span class="line">THIRD_IMAGE=26</span><br><span class="line">CONVOLUTION_NUMBER = 1</span><br><span class="line"></span><br><span class="line">from tensorflow.keras import models</span><br><span class="line"></span><br><span class="line">layer_outputs = [layer.output for layer in model.layers]</span><br><span class="line">activation_model = tf.keras.models.Model(inputs = model.input, outputs = layer_outputs)</span><br><span class="line"></span><br><span class="line">for x in range(0,4):</span><br><span class="line">  f1 = activation_model.predict(test_images[FIRST_IMAGE].reshape(1, 28, 28, 1))[x]</span><br><span class="line">  axarr[0,x].imshow(f1[0, : , :, CONVOLUTION_NUMBER], cmap=&apos;inferno&apos;)</span><br><span class="line">  axarr[0,x].grid(False)</span><br><span class="line">  f2 = activation_model.predict(test_images[SECOND_IMAGE].reshape(1, 28, 28, 1))[x]</span><br><span class="line">  axarr[1,x].imshow(f2[0, : , :, CONVOLUTION_NUMBER], cmap=&apos;inferno&apos;)</span><br><span class="line">  axarr[1,x].grid(False)</span><br><span class="line">  f3 = activation_model.predict(test_images[THIRD_IMAGE].reshape(1, 28, 28, 1))[x]</span><br><span class="line">  axarr[2,x].imshow(f3[0, : , :, CONVOLUTION_NUMBER], cmap=&apos;inferno&apos;)</span><br><span class="line">  axarr[2,x].grid(False)</span><br></pre></td></tr></table></figure></p><ul><li><a href="https://colab.research.google.com/github/lmoroney/dlaicourse/blob/master/Course%201%20-%20Part%206%20-%20Lesson%203%20-%20Notebook.ipynb#scrollTo=kDHjf-ehaBqm" target="_blank" rel="noopener">Details of Convolutions</a></li><li><a href="https://lodev.org/cgtutor/filtering.html" target="_blank" rel="noopener">Image Filtering</a></li></ul><h2 id="Week-4：Human-or-hourse"><a href="#Week-4：Human-or-hourse" class="headerlink" title="Week 4：Human or hourse"></a>Week 4：Human or hourse</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br></pre></td><td class="code"><pre><span class="line">import tensorflow as tf</span><br><span class="line">import os</span><br><span class="line">import zipfile</span><br><span class="line"></span><br><span class="line">DESIRED_ACCURACY = 0.999</span><br><span class="line"></span><br><span class="line">!wget --no-check-certificate \</span><br><span class="line">    &quot;https://storage.googleapis.com/laurencemoroney-blog.appspot.com/happy-or-sad.zip&quot; \</span><br><span class="line">    -O &quot;/tmp/happy-or-sad.zip&quot;</span><br><span class="line"></span><br><span class="line">zip_ref = zipfile.ZipFile(&quot;/tmp/happy-or-sad.zip&quot;, &apos;r&apos;)</span><br><span class="line">zip_ref.extractall(&quot;/tmp/h-or-s&quot;)</span><br><span class="line">zip_ref.close()</span><br><span class="line"></span><br><span class="line">class myCallback(tf.keras.callbacks.Callback):</span><br><span class="line">  def on_epoch_end(self, epoch, logs=&#123;&#125;):</span><br><span class="line">    if(logs.get(&apos;acc&apos;)&gt;DESIRED_ACCURACY):</span><br><span class="line">      print(&quot;\nReached 99.9% accuracy so cancelling training!&quot;)</span><br><span class="line">      self.model.stop_training = True</span><br><span class="line"></span><br><span class="line">callbacks = myCallback()</span><br><span class="line"></span><br><span class="line">model = tf.keras.models.Sequential([</span><br><span class="line">    tf.keras.layers.Conv2D(16, (3,3), activation=&apos;relu&apos;, input_shape=(150, 150, 3)),</span><br><span class="line">    tf.keras.layers.MaxPooling2D(2, 2),</span><br><span class="line">    tf.keras.layers.Conv2D(32, (3,3), activation=&apos;relu&apos;),</span><br><span class="line">    tf.keras.layers.MaxPooling2D(2,2),</span><br><span class="line">    tf.keras.layers.Conv2D(32, (3,3), activation=&apos;relu&apos;),</span><br><span class="line">    tf.keras.layers.MaxPooling2D(2,2),</span><br><span class="line">    tf.keras.layers.Flatten(),</span><br><span class="line">    tf.keras.layers.Dense(512, activation=&apos;relu&apos;),</span><br><span class="line">    tf.keras.layers.Dense(1, activation=&apos;sigmoid&apos;)</span><br><span class="line">])</span><br><span class="line"></span><br><span class="line">from tensorflow.keras.optimizers import RMSprop</span><br><span class="line"></span><br><span class="line">model.compile(loss=&apos;binary_crossentropy&apos;,</span><br><span class="line">              optimizer=RMSprop(lr=0.001),</span><br><span class="line">              metrics=[&apos;acc&apos;])</span><br><span class="line"></span><br><span class="line">from tensorflow.keras.preprocessing.image import ImageDataGenerator</span><br><span class="line"></span><br><span class="line">train_datagen = ImageDataGenerator(rescale=1/255)</span><br><span class="line"></span><br><span class="line">train_generator = train_datagen.flow_from_directory(</span><br><span class="line">        &quot;/tmp/h-or-s&quot;,  </span><br><span class="line">        target_size=(150, 150), </span><br><span class="line">        batch_size=10,</span><br><span class="line">        class_mode=&apos;binary&apos;)</span><br><span class="line"></span><br><span class="line"># Expected output: &apos;Found 80 images belonging to 2 classes&apos;</span><br><span class="line"></span><br><span class="line">history = model.fit_generator(</span><br><span class="line">      train_generator,</span><br><span class="line">      steps_per_epoch=2,  </span><br><span class="line">      epochs=15,</span><br><span class="line">      verbose=1,</span><br><span class="line">      callbacks=[callbacks])</span><br></pre></td></tr></table></figure>]]></content>
      
      
      <categories>
          
          <category> Tensorflow </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
            <tag> Tensorflow </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Python</title>
      <link href="/2018/12/20/python/"/>
      <url>/2018/12/20/python/</url>
      
        <content type="html"><![CDATA[<p>python中的一些语法&amp;函数。</p><ol><li><p>原数+1: <code>x += 1</code></p></li><li><p>字符串与数字的转换：</p></li></ol><ul><li><code>str(2) = &#39;2&#39;</code></li><li><code>int(&#39;2&#39;) = 2</code></li></ul><ol><li>字符串的提取：</li></ol><ul><li><code>str.strip([chars])</code><ul><li>用于移除字符串头尾指定的字符（默认为空格或换行符）或字符序列</li></ul></li><li><code>str.split(str=&quot;&quot;, num=string.count(str))</code><ul><li>通过指定分隔符对字符串进行切片，如果参数 num 有指定值，则分隔 num+1 个子字符串</li></ul></li></ul><ol><li>字符串的拼接：</li></ol><ul><li>用%拼接：<ul><li><code>str = &#39;There are %s, %s, %s on the table.&#39; % (fruit1,fruit2,fruit3)</code></li><li><code>str = &#39;There are %(fruit1)s,%(fruit2)s,%(fruit3)s on the table&#39; % {&#39;fruit1&#39;:fruit1,&#39;fruit2&#39;:fruit2,&#39;fruit3&#39;:fruit3}</code></li><li><code>%s</code> str, <code>%d</code> int, <code>%f</code> float, <code>%.2f</code> 两位小数float, <code>%8s</code> 8位占位符，<code>%-8s</code> 左对齐 </li></ul></li><li><p>用join()拼接：</p>  <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">temp = [&apos;There are &apos;,fruit1,&apos;,&apos;,fruit2,&apos;,&apos;,fruit3,&apos; on the table&apos;]</span><br><span class="line">&apos;&apos;.join(temp)</span><br></pre></td></tr></table></figure></li><li><p>用format()拼接：</p>  <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">str = &apos;There are &#123;2&#125;, &#123;1&#125;, &#123;0&#125; on the table&apos;</span><br><span class="line">str.format(fruit1,fruit2,fruit3) #fruit1出现在0的位置</span><br></pre></td></tr></table></figure>  <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"> str = &apos;There are &#123;fruit1&#125;, &#123;fruit2&#125;, &#123;fruit3&#125; on the table&apos;</span><br><span class="line">str.format(fruit1=fruit1,fruit2=fruit2,fruit3=fruit3)</span><br></pre></td></tr></table></figure></li></ul><ol><li><p>遍历list：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">from collections import Counter, OrderedDict</span><br><span class="line">x = Counter([1,2,2,2,2,3,3,3,4,4,4,4])</span><br><span class="line"></span><br><span class="line"># Counter(&#123;2: 4, 4: 4, 3: 3, 1: 1&#125;)</span><br><span class="line"></span><br><span class="line">sorted(x.items())</span><br><span class="line">x.most_common()</span><br><span class="line">x.most_common(1)</span><br><span class="line">sorted(x, key=x.get, reverse=True)</span><br><span class="line">sorted(x.items(), key=lambda pair: pair[1], reverse=True)</span><br><span class="line"></span><br><span class="line">y = OrderedDict(x.most_common())</span><br></pre></td></tr></table></figure></li><li><p>numpy.random.uniform(low,high,size): <a href="https://blog.csdn.net/u013920434/article/details/52507173" target="_blank" rel="noopener">参考</a><br>low: 采样下界，float类型，默认值为0；<br>high: 采样上界，float类型，默认值为1；<br>size: 输出样本数目，为int或元组(tuple)类型，例如，size=(m,n,k), 则输出m<em>n</em>k个样本，缺省时输出1个值。<br>返回值：ndarray类型，其形状和参数size中描述一致。</p></li><li><p>OrderedDict(): <a href="https://www.cnblogs.com/gide/p/6370082.html" target="_blank" rel="noopener">参考</a><br>很多人认为python中的字典是无序的，因为它是按照hash来存储的，但是python中有个模块collections(英文，收集、集合)，里面自带了一个子类OrderedDict，实现了对字典对象中元素的排序。</p></li><li><p>zip():<a href="http://www.runoob.com/python/python-func-zip.html" target="_blank" rel="noopener">参考</a><br>zip() 函数用于将可迭代的对象作为参数，将对象中对应的元素打包成一个个元组，然后返回由这些元组组成的列表。<br>如果各个迭代器的元素个数不一致，则返回列表长度与最短的对象相同，利用 * 号操作符，可以将元组解压为列表。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">a = [1,2,3]</span><br><span class="line">b = [4,5,6]</span><br><span class="line">c = [4,5,6,7,8]</span><br><span class="line">zipped = zip(a,b)     # 打包为元组的列表</span><br><span class="line"># list(zipped) : [(1, 4), (2, 5), (3, 6)]</span><br><span class="line">zip(a,c)              # 元素个数与最短的列表一致</span><br><span class="line"># [(1, 4), (2, 5), (3, 6)]</span><br><span class="line">zip(*zipped)          # 与 zip 相反，*zipped 可理解为解压，返回二维矩阵式</span><br><span class="line"># [(1, 2, 3), (4, 5, 6)]</span><br></pre></td></tr></table></figure></li><li><p>删除list中的元素</p></li></ol><ul><li><p>remove: 删除单个元素，删除首个符合条件的元素，按值删除<br>举例说明:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">str=[1,2,3,4,5,2,6]</span><br><span class="line">str.remove(2)</span><br><span class="line">str</span><br><span class="line"># [1, 3, 4, 5, 2, 6]</span><br></pre></td></tr></table></figure></li><li><p>pop:  删除单个或多个元素，按位删除(根据索引删除)</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">str=[0,1,2,3,4,5,6]</span><br><span class="line">str.pop(1)   #pop删除时会返回被删除的元素</span><br><span class="line"># 1</span><br><span class="line">str</span><br><span class="line"># [0, 2, 3, 4, 5, 6]</span><br><span class="line">str2=[&apos;abc&apos;,&apos;bcd&apos;,&apos;dce&apos;]</span><br><span class="line">str2.pop(2)</span><br><span class="line"># &apos;dce&apos;</span><br><span class="line">str2</span><br><span class="line"># [&apos;abc&apos;, &apos;bcd&apos;]</span><br></pre></td></tr></table></figure></li><li><p>del：它是根据索引(元素所在位置)来删除</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">str=[1,2,3,4,5,2,6]</span><br><span class="line">del str[1]</span><br><span class="line">str</span><br><span class="line"># [1, 3, 4, 5, 2, 6]</span><br><span class="line">str2=[&apos;abc&apos;,&apos;bcd&apos;,&apos;dce&apos;]</span><br><span class="line">del str2[1]</span><br><span class="line">str2</span><br><span class="line"># [&apos;abc&apos;, &apos;dce&apos;]</span><br></pre></td></tr></table></figure></li></ul><p>除此之外，del还可以删除指定范围内的值。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">str=[0,1,2,3,4,5,6]</span><br><span class="line">del str[2:4]  #删除从第2个元素开始，到第4个为止的元素(但是不包括尾部元素)</span><br><span class="line">str</span><br><span class="line"># [0, 1, 4, 5, 6]</span><br></pre></td></tr></table></figure></p><p>del 也可以删除整个数据对象(列表、集合等)<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">str=[0,1,2,3,4,5,6]</span><br><span class="line">del str</span><br><span class="line">str         #删除后，找不到对象</span><br></pre></td></tr></table></figure></p><ol><li><p><a href="https://blog.csdn.net/quintind/article/details/79850455" target="_blank" rel="noopener">LabelEncoder</a><br>LabelEncoder是用来对分类型特征值进行编码，即对不连续的数值或文本进行编码。其中包含以下常用方法：<br><code>fit(y)</code> ：fit可看做一本空字典，y可看作要塞到字典中的词。<br><code>fit_transform(y)</code>：相当于先进行fit再进行transform，即把y塞到字典中去以后再进行transform得到索引值。<br><code>inverse_transform(y)</code>：根据索引值y获得原始数据。<br><code>transform(y)</code> ：将y转变成索引值。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">&gt;&gt;&gt; le = preprocessing.LabelEncoder()</span><br><span class="line">&gt;&gt;&gt; le.fit([&quot;paris&quot;, &quot;paris&quot;, &quot;tokyo&quot;, &quot;amsterdam&quot;])</span><br><span class="line">LabelEncoder()</span><br><span class="line">&gt;&gt;&gt; list(le.classes_)</span><br><span class="line">[&apos;amsterdam&apos;, &apos;paris&apos;, &apos;tokyo&apos;]</span><br><span class="line">&gt;&gt;&gt; le.transform([&quot;tokyo&quot;, &quot;tokyo&quot;, &quot;paris&quot;]) </span><br><span class="line">array([2, 2, 1]...)</span><br><span class="line">&gt;&gt;&gt; list(le.inverse_transform([2, 2, 1]))</span><br><span class="line">[&apos;tokyo&apos;, &apos;tokyo&apos;, &apos;paris&apos;]</span><br></pre></td></tr></table></figure></li><li><p><a href="https://www.cnblogs.com/chaosimple/p/4153167.html" target="_blank" rel="noopener">数据预处理</a></p></li><li><p><a href="https://www.cnblogs.com/aiguiling/p/8594023.html" target="_blank" rel="noopener">排列组合</a></p></li></ol><ul><li><code>product</code> 笛卡尔积　　（有放回抽样排列）</li><li><code>permutations</code> 排列　　（不放回抽样排列）</li><li><code>combinations</code> 组合,没有重复　　（不放回抽样组合）</li><li><code>combinations_with_replacement</code> 组合,有重复　　（有放回抽样组合）</li><li><p>eg: </p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">import itertools</span><br><span class="line">itertools.product(&apos;ABCD&apos;, repeat = 2)</span><br></pre></td></tr></table></figure></li><li><p>combinations和permutations返回的是对象地址，iterators（迭代器），可以用list()转换为list</p></li></ul><ol><li><p>删除内存<br>del 可以删除多个变量，del a,b,c,d<br>办法：<br>import gc （garbage collector）<br>del a<br>gc.collect()<br>马上内存就释放了。</p></li><li><p><code>enumerate()</code> 函数用于将一个可遍历的数据对象(如列表、元组或字符串)组合为一个索引序列，同时列出数据和数据下标，一般用在 for 循环当中。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">&gt;&gt;&gt;seasons = [&apos;Spring&apos;, &apos;Summer&apos;, &apos;Fall&apos;, &apos;Winter&apos;]</span><br><span class="line">&gt;&gt;&gt; list(enumerate(seasons))</span><br><span class="line">[(0, &apos;Spring&apos;), (1, &apos;Summer&apos;), (2, &apos;Fall&apos;), (3, &apos;Winter&apos;)]</span><br><span class="line">&gt;&gt;&gt; list(enumerate(seasons, start=1))       # 下标从 1 开始</span><br><span class="line">[(1, &apos;Spring&apos;), (2, &apos;Summer&apos;), (3, &apos;Fall&apos;), (4, &apos;Winter&apos;)]</span><br></pre></td></tr></table></figure></li><li><p>将lst = [ [1, 2, 3], [2, 1, 3], [8, 4, 3] ]，变为[1, 2, 3, 2, 1, 3, 8, 4, 3]</p></li></ol><ul><li><code>myList = [x for j in lst for x in j]</code></li><li><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">from itertools import chain</span><br><span class="line">mylist = list(chain(*lst))</span><br></pre></td></tr></table></figure></li></ul><ol><li><p><a href="https://www.cnblogs.com/herbert/p/3468294.html" target="_blank" rel="noopener">namedtuple</a><br>namedtuple创建一个和tuple类似的对象，而且对象拥有可以访问的属性。这对象更像带有数据属性的类，不过数据属性是只读的。<br>eg: <code>TPoint = namedtuple(&#39;TPoint&#39;, [&#39;x&#39;, &#39;y&#39;])</code> 创建一个TPoint类型，而且带有属性x, y.</p></li><li><p><a href="http://www.runoob.com/python/python-func-isinstance.html" target="_blank" rel="noopener">isinstance</a><br>用来判断一个对象是否是一个已知的类型，类似 type()。<br><code>isinstance()</code> 与 <code>type()</code> 区别：</p></li></ol><ul><li><code>type()</code> 不会认为子类是一种父类类型，不考虑继承关系。</li><li><code>isinstance()</code> 会认为子类是一种父类类型，考虑继承关系。<br>使用：<code>isinstance(object, classinfo)</code></li></ul><ol><li><p><code>lambda</code>,<code>map</code>, <code>filter</code>, <code>reduce</code></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br></pre></td><td class="code"><pre><span class="line"># lambda</span><br><span class="line">square = lambda number: number*number</span><br><span class="line"></span><br><span class="line"># map</span><br><span class="line">def inefficientSquare(number):</span><br><span class="line">  result = number*number</span><br><span class="line">  return result</span><br><span class="line">map(inefficientSquare,my_list)</span><br><span class="line"></span><br><span class="line"># map, filter, reduce</span><br><span class="line">number = [1,2,3,4,5,6]</span><br><span class="line">odd_numbers = []</span><br><span class="line">squared_odd_numbers = []</span><br><span class="line">total = 0</span><br><span class="line"></span><br><span class="line"># filter for odd numbers:</span><br><span class="line">for number in numbers:</span><br><span class="line">  if number%2 == 1:</span><br><span class="line">    odd_numbers.append(number)</span><br><span class="line"></span><br><span class="line">odd_numbers = filter(lambda n:n%2==1,numbers)</span><br><span class="line"></span><br><span class="line"># square all odd numbers:</span><br><span class="line">for number in odd_numbers:</span><br><span class="line">  squared_odd_numbers.append(number*number)</span><br><span class="line"></span><br><span class="line">squared_odd_numbers = map(lambda n:n*n,odd_numbers)</span><br><span class="line"></span><br><span class="line"># calculate total</span><br><span class="line">for number in squared_odd_numbers:</span><br><span class="line">  total += number</span><br><span class="line"></span><br><span class="line">total = reduce(lambda acc, n: acc+n, suqared_odd_numbers)</span><br><span class="line"></span><br><span class="line">pd.applymap(lambda x: x.replace(&apos;\&apos;&apos;, &apos;&apos;))</span><br></pre></td></tr></table></figure></li><li><p>加载文件</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">for file in os.listdir(&apos;/fold&apos;):</span><br><span class="line">    filename = os.path.join(domain, file)</span><br><span class="line">    data = np.load(filename)</span><br></pre></td></tr></table></figure></li><li><p><a href="https://www.geeksforgeeks.org/reading-images-in-python/" target="_blank" rel="noopener">加载图片</a></p></li></ol><ul><li><p>opencv</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line">brightness_4</span><br><span class="line"># Python program to read image using OpenCV </span><br><span class="line">  </span><br><span class="line"># importing OpenCV(cv2) module </span><br><span class="line">import cv2 </span><br><span class="line">  </span><br><span class="line"># Save image in set directory </span><br><span class="line"># Read RGB image </span><br><span class="line">img = cv2.imread(&apos;g4g.png&apos;)  </span><br><span class="line">  </span><br><span class="line"># Output img with window name as &apos;image&apos; </span><br><span class="line">cv2.imshow(&apos;image&apos;, img)  </span><br><span class="line">  </span><br><span class="line"># Maintain output window utill </span><br><span class="line"># user presses a key </span><br><span class="line">cv2.waitKey(0)         </span><br><span class="line">  </span><br><span class="line"># Destroying present windows on screen </span><br><span class="line">cv2.destroyAllWindows()</span><br></pre></td></tr></table></figure></li><li><p>matplot</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"># Python program to read  </span><br><span class="line"># image using matplotlib </span><br><span class="line">  </span><br><span class="line"># importing matplotlib modules </span><br><span class="line">import matplotlib.image as mpimg </span><br><span class="line">import matplotlib.pyplot as plt </span><br><span class="line">  </span><br><span class="line"># Read Images </span><br><span class="line">img = mpimg.imread(&apos;g4g.png&apos;) </span><br><span class="line">  </span><br><span class="line"># Output Images </span><br><span class="line">plt.imshow(img)</span><br></pre></td></tr></table></figure></li><li><p>pil</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"># Python program to read </span><br><span class="line"># image using PIL module </span><br><span class="line">  </span><br><span class="line"># importing PIL </span><br><span class="line">from PIL import Image </span><br><span class="line">  </span><br><span class="line"># Read image </span><br><span class="line">img = Image.open(&apos;g4g.png&apos;) </span><br><span class="line">  </span><br><span class="line"># Output Images </span><br><span class="line">img.show() </span><br><span class="line">  </span><br><span class="line"># prints format of image </span><br><span class="line">print(img.format) </span><br><span class="line">  </span><br><span class="line"># prints mode of image </span><br><span class="line">print(img.mode)</span><br></pre></td></tr></table></figure></li><li><p>加载多张图片</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">plt.figure(figsize=(50,50))</span><br><span class="line"></span><br><span class="line">for i in range(len(files)):</span><br><span class="line">    face = files[i]</span><br><span class="line">    filename = &apos;./Faces128/eface_%s.png&apos; %face</span><br><span class="line">    img = mpimg.imread(filename) </span><br><span class="line">    plt.subplot(10,1,i+1)</span><br><span class="line">    plt.imshow(img) </span><br><span class="line">    plt.title(ranks[i])</span><br></pre></td></tr></table></figure></li><li><p>保存<br><code>numpy.save(file, arr, allow_pickle=True, fix_imports=True)</code></p></li><li><p>张量扩充<br><code>np.tile(a,[5,1])</code></p></li><li><p>矩阵上三角<br><code>at = np.triu(am, 0)</code></p></li><li><p>按顺序set</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">def ordered_set(x):</span><br><span class="line">    y = list(set(x))</span><br><span class="line">    y.sort(key=x.index)</span><br><span class="line">    return y</span><br><span class="line"></span><br><span class="line">hadm_icd9_3s = list(map(ordered_set,hadm_icd9_3[&apos;ICD9_3&apos;].values))</span><br></pre></td></tr></table></figure></li><li><p>排序</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"># 反转</span><br><span class="line">x.reverse() </span><br><span class="line"></span><br><span class="line"># 排序</span><br><span class="line">x.sort()</span><br><span class="line">x.sort(reverse=True) # 逆序</span><br><span class="line"></span><br><span class="line"># 排序不覆盖源列表</span><br><span class="line">b = sorted(x)</span><br><span class="line">b = sorted(x, reverse=True) </span><br><span class="line"></span><br><span class="line"># 返回索引</span><br><span class="line">np.argsort(x)</span><br><span class="line">np.flip(np.argsort(x))</span><br></pre></td></tr></table></figure></li><li><p>读取文件夹下多个文件</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">for file in os.listdir(&apos;./filefold&apos;):</span><br><span class="line">    filename = os.path.join(&apos;./filefold&apos;, file)</span><br><span class="line">    # print(filename)</span><br><span class="line">    data = pd.read_csv(filename, header=None)</span><br></pre></td></tr></table></figure></li><li><p>读取与存储<code>.mat</code>文件</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">import scipy.io as scio</span><br><span class="line"></span><br><span class="line"># 读取</span><br><span class="line">dataFile = &apos;data.mat&apos;</span><br><span class="line">data = scio.loadmat(dataFile)</span><br><span class="line"></span><br><span class="line">type(data) #查看，data为字典格式</span><br><span class="line"></span><br><span class="line"># 存储</span><br><span class="line">scio.savemat(dataFile, &#123;&apos;data&apos;:data&#125;)</span><br></pre></td></tr></table></figure></li></ul>]]></content>
      
      
      <categories>
          
          <category> Python </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Python </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Theano</title>
      <link href="/2018/12/19/Theano/"/>
      <url>/2018/12/19/Theano/</url>
      
        <content type="html"><![CDATA[<p>在看<a href="https://github.com/mp2893/gram">GRAM</a>源码，记录一下Theano的语法。参考：<a href="https://morvanzhou.github.io/tutorials/machine-learning/theano/" target="_blank" rel="noopener">Theano教程系列</a>，<a href="http://deeplearning.net/software/theano/library/tensor/index.html" target="_blank" rel="noopener">Theano API documentation</a>, <a href="https://www.cnblogs.com/shouhuxianjian/category/699462.html" target="_blank" rel="noopener">Theano教程</a></p><ol><li><p>加载theano和numpy模块，并创建<code>function</code>:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">import numpy as np</span><br><span class="line">import theano.tensor as T</span><br><span class="line">from theano import function</span><br></pre></td></tr></table></figure></li><li><p>定义常量(scalar)及函数(function)：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">x = T.dscalar(&apos;x&apos;)</span><br><span class="line">y = T.dscalar(&apos;y&apos;)</span><br><span class="line">z = x+y</span><br><span class="line"></span><br><span class="line">f = function([x,y],z)</span><br><span class="line"></span><br><span class="line">print(f(2,3))</span><br><span class="line"># 5.0</span><br></pre></td></tr></table></figure></li><li><p>打印原函数</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">from theano import pp</span><br><span class="line"></span><br><span class="line">print(pp(z))</span><br><span class="line"># (x+y)</span><br></pre></td></tr></table></figure></li><li><p>定义矩阵</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">x = T.dmatrix(&apos;x&apos;)</span><br><span class="line">y = T.dmatrix(&apos;y&apos;)</span><br><span class="line">z = x+y</span><br><span class="line"></span><br><span class="line">printf(</span><br><span class="line">f(np.arange(12).reshape(3,4),10*np.ones(3,4))</span><br><span class="line">)</span><br><span class="line"></span><br><span class="line">&apos;&apos;&apos;</span><br><span class="line">[[ 10.  11.  12.  13.]</span><br><span class="line"> [ 14.  15.  16.  17.]</span><br><span class="line"> [ 18.  19.  20.  21.]]</span><br><span class="line"> &apos;&apos;&apos;</span><br></pre></td></tr></table></figure></li><li><p>Function的用法</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">import numpy as np</span><br><span class="line">import theano.tensor as T</span><br><span class="line">import theano</span><br></pre></td></tr></table></figure></li></ol><ul><li><p>激活函数</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">x = T.dmatrix(&apos;x&apos;)</span><br><span class="line">s = 1/(1+T.exp(-x))</span><br><span class="line"></span><br><span class="line">logistic = theano.function([x],s)</span><br><span class="line"></span><br><span class="line">print(logistic([[0,1],[-2,-3]]))</span><br><span class="line"></span><br><span class="line">&apos;&apos;&apos;</span><br><span class="line">[[ 0.5         0.73105858]</span><br><span class="line"> [ 0.26894142  0.11920292]]</span><br></pre></td></tr></table></figure></li><li><p>多输入/输出</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">a,b = T.dmatrices(&apos;a&apos;,&apos;b&apos;)</span><br><span class="line"></span><br><span class="line">diff = a-b</span><br><span class="line">abs_diff = abs(a-b)</span><br><span class="line"></span><br><span class="line">f = theano.function([a,b],[diff,abs_diff])</span><br><span class="line"></span><br><span class="line">x1,x2= f(</span><br><span class="line">    np.ones((2,2)), # a</span><br><span class="line">    np.arange(4).reshape((2,2))  # b</span><br><span class="line">)</span><br><span class="line"></span><br><span class="line">&quot;&quot;&quot;</span><br><span class="line">array([[ 1.,  0.],</span><br><span class="line">       [-1., -2.]]),</span><br><span class="line">array([[ 1.,  0.],</span><br><span class="line">       [ 1.,  2.]]),  </span><br><span class="line">&quot;&quot;&quot;</span><br></pre></td></tr></table></figure></li><li><p>默认值 &amp; 名字</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line">x,y,w = T.dscalars(&apos;x&apos;,&apos;y&apos;,&apos;w&apos;)</span><br><span class="line">z = (x+y)*w</span><br><span class="line"></span><br><span class="line">f = theano.function([x,</span><br><span class="line">                     theano.In(y,value=1),</span><br><span class="line">                     theano.In(w,value=2)],</span><br><span class="line">                    z)</span><br><span class="line"></span><br><span class="line">print(f(23))    # 使用默认</span><br><span class="line">print(f(23,1,4)) # 不使用默认</span><br><span class="line">&quot;&quot;&quot;</span><br><span class="line">48.0</span><br><span class="line">100.0</span><br><span class="line">&quot;&quot;&quot;</span><br><span class="line"></span><br><span class="line">f = theano.function([x,</span><br><span class="line">                     theano.In(y,value=1),</span><br><span class="line">                     theano.In(w,value=2,name=&apos;weights&apos;)],</span><br><span class="line">                    z)</span><br><span class="line">                    </span><br><span class="line">print (f(23,1,weights=4)) ##调用方式</span><br><span class="line"></span><br><span class="line">&quot;&quot;&quot;</span><br><span class="line">100.0</span><br><span class="line">&quot;&quot;&quot;</span><br></pre></td></tr></table></figure></li><li><p>Shared变量</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">state = theano.shared(np.array(0,dtype=np.float64), &apos;state&apos;)     # inital state = 0</span><br><span class="line">inc = T.scalar(&apos;inc&apos;, dtype=state.dtype)</span><br><span class="line">accumulator = theano.function([inc], state, updates=[(state, state+inc)]) # updates: state+=inc</span><br><span class="line"></span><br><span class="line"># to get variable value</span><br><span class="line">print(state.get_value())</span><br><span class="line"># 0.0</span><br><span class="line"></span><br><span class="line">accumulator(1)   # return previous value, 0 in here</span><br><span class="line">print(state.get_value())</span><br><span class="line"># 1.0</span><br><span class="line"></span><br><span class="line">accumulator(10)  # return previous value, 1 in here</span><br><span class="line">print(state.get_value())</span><br><span class="line"># 11.0</span><br><span class="line"></span><br><span class="line">state.set_value(-1)</span><br><span class="line">accumulator(3)</span><br><span class="line">print(state.get_value())</span><br><span class="line"># 2.0</span><br></pre></td></tr></table></figure></li><li><p><a href="http://deeplearning.net/software/theano/library/tensor/nnet/nnet.html" target="_blank" rel="noopener">激活函数</a>：<br><code>theano.tensor.nnet.nnet.sigmoid(x)</code><br><code>softplus()</code>, <code>relu()</code>, <code>softmax()</code>,<code>tanh()</code></p></li><li><p>Layer类</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">l1 = Layer(inputs, in_size, out_size, T.nnet.relu)</span><br><span class="line">l2 = Layer(l1.outputs, in_size, out_size, None) # None采用默认的线性激活函数</span><br><span class="line"></span><br><span class="line">class Layer(object):</span><br><span class="line">    def __init__(self, inputs, in_size, out_size, activation_function=None):</span><br><span class="line">        self.W = theano.shared(np.random.normal(0, 1, (in_size, out_size)))</span><br><span class="line">        self.b = theano.shared(np.zeros((out_size, )) + 0.1)</span><br><span class="line">        self.Wx_plus_b = T.dot(inputs, self.W) + self.b</span><br><span class="line">        self.activation_function = activation_function</span><br><span class="line">        if activation_function is None:</span><br><span class="line">            self.outputs = self.Wx_plus_b</span><br><span class="line">        else:</span><br><span class="line">            self.outputs = self.activation_function(self.Wx_plus_b)</span><br></pre></td></tr></table></figure></li><li><p>数据类型<br>x=T.scalar(‘myvar’,dtype=theano.config.floatX)#创建0维阵列<br>x=T.vector(‘myvar’,dtype=theano.config.floatX)#创建以为阵列<br>x=T.row(‘myvar’,dtype=theano.config.floatX)#创建二维阵列,行数为1<br>x=T.col(‘myvar’,dtype=theano.config.floatX)#创建二维阵列,列数为1<br>x=T.matrix(‘myvar’,dtype=theano.config.floatX)#创建二维矩阵<br>x=T.tensor3(‘myvar’,dtype=theano.config.floatX)#创建三维张量<br>x=T.tensor4(‘myvar’,dtype=theano.config.floatX)#创建四维张量<br>x=T.tensor5(‘myvar’,dtype=theano.config.floatX)#创建五维张量<br>x.ndim#输出维度看看</p></li></ul>]]></content>
      
      
      <categories>
          
          <category> Machine Learning </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
            <tag> Theano </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>Deep EHR Survey</title>
      <link href="/2018/12/18/EHRSurvey/"/>
      <url>/2018/12/18/EHRSurvey/</url>
      
        <content type="html"><![CDATA[<p>This post is the reading notes of paper [1], because I’d like to learn some machine learning work on EHR recently.</p><h1 id="EHR-Information-Extraction-IE"><a href="#EHR-Information-Extraction-IE" class="headerlink" title="EHR Information Extraction(IE)"></a>EHR Information Extraction(IE)</h1><p>Each patient encounter is associated with several clinical notes, such as admission notes, discharge summaries, and transfer orders.</p><h3 id="Single-concepte-extraction"><a href="#Single-concepte-extraction" class="headerlink" title="Single concepte extraction"></a>Single concepte extraction</h3><ul><li>The most fundamental task involving clinical free text is the extraction of structured medical conceptes, such as diseases, treatments, or procedures.<ul><li>Jagannatha et al.[2,3] treat the concept extraction problem as a sequence labeling task. They experiment with several deep architectures based on RNNs, including LSTMs and GRUs, bidirectional LSTMs(Bi-LSTM), and various combinatons of LSTMs with traditional CRF.</li></ul></li><li>Other applications of deep learning to clinical concept extraction include named entity recognition(NER).<ul><li>Wu et al.[4] apply pre-trained word embeddings on Chinese clinical text using a CNN.</li></ul></li></ul><h3 id="Temporal-Event-Extraction"><a href="#Temporal-Event-Extraction" class="headerlink" title="Temporal Event Extraction"></a>Temporal Event Extraction</h3><ul><li>This subtask tackles the most complex issue of assigning notions of time to each extracted EHR concept.<ul><li>Fries[5] devised a framework to extract medical events and their corresponding times from clinical notes using a standard RNN.</li><li>Fries[6] utilizes Stanford’s DeepDive application for structured relationships and predictions.</li></ul></li></ul><h3 id="Relation-Extraction"><a href="#Relation-Extraction" class="headerlink" title="Relation Extraction"></a>Relation Extraction</h3><ul><li>Relation extraction deals with structured relationships between medical concepts in free text, including relations such as <em>treatment X improves/worsens/causes condition Y</em>, or <em>test X reveals medical probem Y</em>.<ul><li>Lv et al.[7] use standard text pre-processing methods and UMLS-based word-to-concept mappings in conjuncton with sparse autoencoders to generate features for input to a CRF classifier.</li></ul></li></ul><h3 id="Abbreviation-Expansion"><a href="#Abbreviation-Expansion" class="headerlink" title="Abbreviation Expansion"></a>Abbreviation Expansion</h3><ul><li>Each abbreviation can bave tens of possible explanations.<ul><li>Liu et al.[[8]] tackle the roblem by utilizing word embedding approaches.</li></ul></li></ul><h1 id="EHR-Representation-Learning"><a href="#EHR-Representation-Learning" class="headerlink" title="EHR Representation Learning"></a>EHR Representation Learning</h1><p>Recent deep learning approaches have been used to project discrete codes into vector space for more detailed analysis and more precise predictive tasks.</p><h3 id="Concept-Representation"><a href="#Concept-Representation" class="headerlink" title="Concept Representation"></a>Concept Representation</h3><p>Several recent studies have applied deep unsupervsed representation learning techniques to derive EHR concept vectors that capture the latent similairities and natural clusters between medical concepts.</p><ul><li>Distributed Embedding: Several researchers have applied NLP techniques for summarizing sparse medical codes into a fixed-size and compressed vector format, such as skip-gram[9,10,13]</li><li>Latent Encoding: Aside from NLP-inspired methods, other common deep learning representation learning techniques have also been used for representing EHR concepts, such as RBM[11] and AE[7].</li></ul><h3 id="Patient-Representation"><a href="#Patient-Representation" class="headerlink" title="Patient Representation"></a>Patient Representation</h3><p>Several different deep learning methods for obtaining vector representations of patients have been proposed in [11,13,14,15,16].</p><h1 id="Outcome-Prediction"><a href="#Outcome-Prediction" class="headerlink" title="Outcome Prediction"></a>Outcome Prediction</h1><ul><li>Static Outcome Prediction: The prediction of a certain outcome without consider temporal constraints.<ul><li>Nguyen et al.[9] use distributed representations and several ANN and linear models to predict heart failure.</li><li>Tran et al[11] derive patient vectors with their modified RBM architecture, then train a logistic regression classifier for suicide risk stratification. </li><li>DeepPatient[2] generated patient vectors with a 3-layer autoencoder, then used these vectors with logistic regression classifiers to predict a wide variety of ICD9-based disease diagnoses within a prediction window.</li><li>Jacobson et al.[16] compared deep unsupervised representation of clinical notes for predicting healthcare-associated infections(HAI), utilizing stacked sparse AEs and stacked RBMs along with a word2vec-based embedding approach.</li><li>Li et al.[17] used a two-layer DBN for identifying osteoporosis.</li></ul></li><li>Temporal Outcome Prediction: Either to predict the outcome or onset within a certain time interval or to make a prediction based on time series data.<ul><li>Cheng et al.[18] trained a CNN on temporal matrices of medical codes per patient for predicting the onset of both congestive heart failure(CHF) and chronic obstructive pulmonary disease(COPD).</li><li>Lipton et al.[19] used LSTM networks for predicting on of 128 diagnoses, using target replication at each time step along with auxiliary targets for less-common diagnostic labels as a form of regularization.</li><li>Choi et al.’s Doctor AI[13] framework was constructed to model how physicians behave by predicting future disease diagnosis along with corresponding timed medication interventions.</li><li>Pham et al.’s DeepCare[20] framework also derives clinical concept vectors via a skip-gram embedding approach.</li><li>Nickerson et al.[21] forecast postoperative responses including postoperative urinary retention (POUR) and temporal patterns of postoperative pain using MLP and LSTM networks to suggest more effective postoperative pain management.</li><li>Nguyen el al.’s Deepr[9] system uses a CNN for predicting unplanned readmission following discharge.</li><li>Esteban et al.[22] used deep models for predicting the onset of complications relating to kidney transplantation.</li><li>Che et al.[23] develop a variation of the recurrent GRU cell(GRU-D) which attempts to better handle missing values in clinical time series</li></ul></li></ul><h1 id="Computational-Phenotyping"><a href="#Computational-Phenotyping" class="headerlink" title="Computational Phenotyping"></a>Computational Phenotyping</h1><p>To revisit and refine broad ilness and diagnosis definitions and boundaries.</p><ul><li>New Phenotype Discovery</li><li>Improving Existing Definitions</li></ul><h1 id="Clinical-Data-De-Identification"><a href="#Clinical-Data-De-Identification" class="headerlink" title="Clinical Data De-Identification"></a>Clinical Data De-Identification</h1><h1 id="Reference"><a href="#Reference" class="headerlink" title="Reference"></a>Reference</h1><ol><li>Shickel B, Tighe P J, Bihorac A, et al. <a href="https://arxiv.org/pdf/1706.03446.pdf" target="_blank" rel="noopener">Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis[J]</a>. IEEE journal of biomedical and health informatics, 2018, 22(5): 1589-1604.</li><li>A. N. Jagannatha and H. Yu, <a href="https://arxiv.org/pdf/1608.00612.pdf" target="_blank" rel="noopener">Structured prediction models for RNN based sequence labeling in clinical text,</a> in EMNLP, 2016</li><li>A. Jagannatha and H. Yu, <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5119627/pdf/nihms823967.pdf" target="_blank" rel="noopener">Bidirectional Recurrent Neural Networks for Medical Event Detection in Electronic Health Records</a>, arXiv, pp. 473–482, 2016.</li><li>Y. Wu, M. Jiang, J. Lei, and H. Xu, <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4624324/pdf/nihms-708181.pdf" target="_blank" rel="noopener">Named Entity Recognition in Chinese Clinical Text Using Deep Neural Network</a>, in Studies in Health Technology and Informatics, vol. 216, 2015, pp. 624–628</li><li>J. A. Fries, <a href="http://www.aclweb.org/anthology/S16-1198" target="_blank" rel="noopener">Brundlefly at SemEval-2016 Task 12: Recurrent Neural Networks vs. Joint Inference for Clinical Temporal Information Extraction</a>, in Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), 2016, pp. 1274–1279.</li><li>C. De Sa, A. Ratner, C. Re, J. Shin, F. Wang, S. Wu, and C. Zhang, <a href="http://www.vldb.org/pvldb/vol8/p1310-shin.pdf" target="_blank" rel="noopener">Incremental knowledge base construction using deepdive</a>, The VLDB Journal, pp. 1–25, 2016.</li><li>X. Lv, Y. Guan, J. Yang, and J. Wu, <a href="https://pdfs.semanticscholar.org/7fac/52a9b0f96fcee6972cc6ac4687068442aee8.pdf" target="_blank" rel="noopener">Clinical Relation Extraction with Deep Learning</a>, International Journal of Hybrid Information Technology, vol. 9, no. 7, pp. 237–248, 2016</li><li>Y. Liu, T. Ge, K. S. Mathews, H. Ji, D. L. Mcguinness, and C. Science, <a href="http://aclweb.org/anthology/W15-3810" target="_blank" rel="noopener">Exploiting Task-Oriented Resources to Learn Word Embeddings for Clinical Abbreviation Expansion</a>, in Proceedings of the 2015 Workshop on Biomedical Natural Language Processing (BioNLP 2015), 2015, pp. 92–97.</li><li>P. Nguyen, T. Tran, N. Wickramasinghe, and S. Venkatesh, <a href="https://arxiv.org/pdf/1607.07519.pdf" target="_blank" rel="noopener">Deepr: A Convolutional Net for Medical Records</a>, arXiv, pp. 1–9, 2016.</li><li>Y. Choi, C. Y.-I. Chiu, and D. Sontag, “Learning Low-Dimensional Representations of Medical Concepts Methods Background,” in AMIA Summit on Clinical Research Informatics, 2016, pp. 41–50.</li><li>D. Ravi, C. Wong, F. Deligianni, M. Berthelot, J. A. Perez, B. Lo, and G.-Z. Yang, “Deep learning for health informatics,” IEEE Journal of Biomedical and Health Informatics, 2016.</li><li>E. Choi, M. T. Bahadori, and J. Sun, <a href="http://nematilab.info/bmijc/assets/170607_paper.pdf" target="_blank" rel="noopener">Doctor AI: Predicting Clinical Events via Recurrent Neural Networks</a>, arXiv, pp. 1–12, 2015.</li><li>T. Tran, T. D. Nguyen, D. Phung, and S. Venkatesh, <a href="https://core.ac.uk/download/pdf/82350634.pdf" target="_blank" rel="noopener">Learning vector representation of medical objects via EMR-driven nonnegative restricted Boltzmann machines (eNRBM)</a>, Journal of Biomedical Informatics,vol. 54, pp. 96–105, 2015.</li><li>D. Ravi, C. Wong, F. Deligianni, M. Berthelot, J. A. Perez, B. Lo, and G.-Z. Yang, <a href="https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7801947" target="_blank" rel="noopener">Deep learning for health informatics</a>, IEEE Journal of Biomedical and Health Informatics, 2016.</li><li>E. Choi, A. Schuetz, W. F. Stewart, and J. Sun, “Using recurrent neural network models for early detection of heart failure onset.” Journal of the American Medical Informatics Association : JAMIA, vol. 292, no. 3, pp. 344–350, 2016.</li><li>O. Jacobson and H. Dalianis, “Applying deep learning on electronic health records in Swedish to predict healthcare-associated infections,” ACL 2016, p. 191, 2016.</li><li>H. Li, X. Li, M. Ramanathan, and A. Zhang, “Identifying informative risk factors and predicting bone disease progression via deep belief networks,” Methods, vol. 69, no. 3, pp. 257–265, 2014.</li><li>Y. Cheng, F. Wang, P. Zhang, H. Xu, and J. Hu, “Risk Prediction with Electronic Health Records : A Deep Learning Approach,” in SIAM International Conference on Data Mining SDM16, 2015.</li><li>Z. C. Lipton, D. C. Kale, C. Elkan, and R. Wetzell, <a href="https://arxiv.org/pdf/1511.03677.pdf" target="_blank" rel="noopener">Learning to Diagnose with LSTM Recurrent Neural Networks</a>, ICLR, 2016.</li><li>T. Pham, T. Tran, D. Phung, and S. Venkatesh, <a href="https://arxiv.org/pdf/1602.00357.pdf" target="_blank" rel="noopener">DeepCare: A Deep Dynamic Memory Model for Predictive Medicine</a>, arXiv, no. i, pp. 1–27, 2016.</li><li>P. Nickerson, P. Tighe, B. Shickel, and P. Rashidi, “Deep neural network architectures for forecasting analgesic response,” in Engineering in Medicine and Biology Society (EMBC), 2016 IEEE 38th Annual International Conference of the. IEEE, 2016, pp. 2966–2969.</li><li>C. Esteban, O. Staeck, Y. Yang, and V. Tresp, “Predicting Clinical Events by Combining Static and Dynamic Information Using Recurrent Neural Networks,” arXiv, 2016.</li><li>Z. Che, S. Purushotham, K. Cho, D. Sontag, and Y. Liu,<a href="https://arxiv.org/pdf/1606.01865.pdf" target="_blank" rel="noopener">Recurrent neural networks for multivariate time series with missing values</a>, arXiv preprint arXiv:1606.01865, 2016.</li></ol>]]></content>
      
      
      <categories>
          
          <category> AI for Medicine </category>
          
      </categories>
      
      
        <tags>
            
            <tag> Machine Learning </tag>
            
            <tag> Medicine </tag>
            
        </tags>
      
    </entry>
    
    
    
    <entry>
      <title>用github+hexo搭建自己的博客</title>
      <link href="/2018/12/17/createBlog/"/>
      <url>/2018/12/17/createBlog/</url>
      
        <content type="html"><![CDATA[<p>整个过程主要是参考了<a href="https://www.cnblogs.com/jackyroc/p/7681938.html" target="_blank" rel="noopener">我是如何利用Github Pages搭建起我的博客，细数一路的坑</a>。但是在搭建的过程中，还是遇到了很多问题，所以写个笔记记录一下。</p><h3 id="一-创建Github-Pages"><a href="#一-创建Github-Pages" class="headerlink" title="一. 创建Github Pages"></a>一. 创建Github Pages</h3><p>这一步可以参考<a href="https://www.cnblogs.com/jackyroc/p/7681938.html" target="_blank" rel="noopener">原文</a></p><h3 id="二-安装Hexo"><a href="#二-安装Hexo" class="headerlink" title="二. 安装Hexo"></a>二. 安装Hexo</h3><ol><li>首先安装<a href="https://nodejs.org/en/download/" target="_blank" rel="noopener">Node.js</a>和<a href="https://git-scm.com/download/" target="_blank" rel="noopener">Git</a></li><li>使用Git Bash安装Hexo:<br><code>npm install hexo-cli -g</code><br><code>hexo init blog //初始化网站，文件夹名为“blog”</code><br><code>npm install</code><br><code>hexo g //hexo generate, 生成静态页面</code><br><code>hexo s //hexo server,启动本地服务器，这一步之后就可以通过http://localhost:4000查看</code></li><li>创建文章<br><code>hexo new &#39;post&#39; //post为文章名字</code></li><li>创建页面<br><code>hexo new &#39;page&#39; //page为页面名字</code></li><li>添加主题<br>以’yilia’为例，先下载主题，存放到themes文件夹下:<br><code>hexo clean</code><br><code>git clone https://github.com/litten/hexo-theme-yilia.git themes/yilia</code><br>找到目录下的_config.yml 文件,打开找到 theme：属性并设置为主题的名字（yilia)。<br>更多主题可以在<a href="https://hexo.io/themes/" target="_blank" rel="noopener">https://hexo.io/themes/</a>上面找</li><li>部署到github上<br><code>$ npm install hexo-deployer-git --save</code><br><code>hexo d</code><br>生成ssh可以参考:<a href="https://blog.csdn.net/weixin_41299929/article/details/79599884" target="_blank" rel="noopener">搭建教程</a></li><li><a href="http://huanyouchen.github.io/2019/03/22/add-tags-pages-to-hexo-maupassant-theme/" target="_blank" rel="noopener">标签页面</a></li><li>语言：修改<code>_config.yml</code>文件中的<code>language:zh-CN</code></li><li>留言板<br>尝试了disqus和gitalk，都不太好用，还是<a href="https://valine.js.org/" target="_blank" rel="noopener">valine</a>方便。按照官网上的流程申请一个app,然后配置一下<code>_config.yml</code>文件就好了</li><li>关于文章的(可选项)[<a href="https://codingpub.github.io/2016/04/10/%E5%AE%9A%E5%88%B6Hexo-maupassant/" target="_blank" rel="noopener">https://codingpub.github.io/2016/04/10/%E5%AE%9A%E5%88%B6Hexo-maupassant/</a>]</li><li>配置主题<br>换了<a href="https://www.haomwei.com/technology/maupassant-hexo.html" target="_blank" rel="noopener">maupassant</a>主题，这个主题简洁，支持latex，很不错。关于主题配置，可以参考<a href="https://cirry.cn/hexo/theme-maupassant-config/#29" target="_blank" rel="noopener">hexo配置主题</a>，感谢菜小牛的答复，终于解决了在info下的外链修改。</li></ol>]]></content>
      
      
      <categories>
          
          <category> Notes </category>
          
      </categories>
      
      
        <tags>
            
            <tag> hexo </tag>
            
        </tags>
      
    </entry>
    
    
  
  
</search>