<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet href="/rss.xsl" type="text/xsl"?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/" version="2.0"><channel><generator>Typlog 3.1 (https://typlog.com)</generator><title><![CDATA[Messense Lv]]></title><description><![CDATA[Yet another blog]]></description><link>https://messense.me/posts/</link><copyright><![CDATA[Copyright 2017 Messense Lv]]></copyright><atom:link href="https://messense.me/feed/post.xml" rel="self" type="application/rss+xml"/><atom:link href="https://pubsubhubbub.appspot.com/" rel="hub"/><pubDate>Mon, 13 Apr 2026 23:48:33 +0000</pubDate><item><title><![CDATA[Run cargo test with valgrind]]></title><guid>https://messense.me/cargo-test-valgrind</guid><link>https://messense.me/cargo-test-valgrind</link><description><![CDATA[Detect memory errors with Valgrind for Rust programs]]></description><dc:creator><![CDATA[messense]]></dc:creator><pubDate>Sun, 28 Aug 2022 08:37:20 +0000</pubDate><content:encoded><![CDATA[<p>As a memory safe language, memory errors are often caught by the Rust compiler in compile time, but when using FFI to interface with C/C++, you often need to use <code>unsafe</code> which can introduce memory unsafety. <a href="https://valgrind.org/">Valgrind</a> is a valuable tool for working with unsafe Rust, the <a href="https://nnethercote.github.io/2022/01/05/rust-and-valgrind.html">Rust and Valgrind</a> post by Nicholas Nethercote is a great introduction if you are new to Valgrind.</p>
<p>There are several ways to use Valgrind with Rust, we'll explore them in this post. First let's create an example Rust project with code that leaks memory</p>
<div class="block-code" data-language="bash"><div class="highlight"><pre><span></span><code><div class="line">$ cargo new example
</div><div class="line">     Created binary <span class="o">(</span>application<span class="o">)</span> <span class="sb">`</span>example<span class="sb">`</span> package
</div><div class="line">$ <span class="nb">cd</span> example
</div></code></pre></div>
</div>
<p>Replace <code>src/main.rs</code> with the following code</p>
<div class="block-code" data-language="rust"><div class="highlight"><pre><span></span><code><div class="line"><span class="k">fn</span> <span class="nf">leak</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
</div><div class="line"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="fm">vec!</span><span class="p">[</span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="mi">1024</span><span class="p">];</span><span class="w"></span>
</div><div class="line"><span class="w">    </span><span class="n">std</span>::<span class="n">mem</span>::<span class="n">forget</span><span class="p">(</span><span class="n">data</span><span class="p">);</span><span class="w"></span>
</div><div class="line"><span class="p">}</span><span class="w"></span>
</div><div class="line">
</div><div class="line"><span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
</div><div class="line"><span class="w">    </span><span class="n">leak</span><span class="p">();</span><span class="w"></span>
</div><div class="line"><span class="p">}</span><span class="w"></span>
</div><div class="line">
</div><div class="line"><span class="cp">#[cfg(test)]</span><span class="w"></span>
</div><div class="line"><span class="k">mod</span> <span class="nn">test</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
</div><div class="line"><span class="w">    </span><span class="k">use</span><span class="w"> </span><span class="k">super</span>::<span class="n">leak</span><span class="p">;</span><span class="w"></span>
</div><div class="line">
</div><div class="line"><span class="w">    </span><span class="cp">#[test]</span><span class="w"></span>
</div><div class="line"><span class="w">    </span><span class="k">fn</span> <span class="nf">test_leak</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
</div><div class="line"><span class="w">        </span><span class="n">leak</span><span class="p">();</span><span class="w"></span>
</div><div class="line"><span class="w">    </span><span class="p">}</span><span class="w"></span>
</div><div class="line"><span class="p">}</span><span class="w"></span>
</div></code></pre></div>
</div>
<h2>Run Valgrind directly</h2>
<p>You can just use Valgrind directly on compiled Rust programs, it's quite easy</p>
<div class="block-code" data-language="bash"><div class="highlight"><pre><span></span><code><div class="line">$ cargo build
</div><div class="line">   Compiling example v0.1.0 <span class="o">(</span>/root/code/example<span class="o">)</span>
</div><div class="line">    Finished dev <span class="o">[</span>unoptimized + debuginfo<span class="o">]</span> target<span class="o">(</span>s<span class="o">)</span> <span class="k">in</span> <span class="m">1</span>.58s
</div><div class="line">$ valgrind target/debug/example
</div><div class="line"><span class="o">==</span><span class="nv">1127561</span><span class="o">==</span> Memcheck, a memory error <span class="nv">detector</span>
</div><div class="line"><span class="o">==</span><span class="nv">1127561</span><span class="o">==</span> Copyright <span class="o">(</span>C<span class="o">)</span> <span class="m">2002</span>-2022, and GNU GPL<span class="err">&#39;</span>d, by Julian Seward et al.
</div><div class="line"><span class="o">==</span><span class="nv">1127561</span><span class="o">==</span> Using Valgrind-3.19.0 and LibVEX<span class="p">;</span> rerun with -h <span class="k">for</span> copyright <span class="nv">info</span>
</div><div class="line"><span class="o">==</span><span class="nv">1127561</span><span class="o">==</span> Command: target/debug/example
</div><div class="line"><span class="o">==</span><span class="nv">1127561</span><span class="o">==</span>
</div><div class="line"><span class="o">==</span><span class="nv">1127561</span><span class="o">==</span>
</div><div class="line"><span class="o">==</span><span class="nv">1127561</span><span class="o">==</span> HEAP SUMMARY:
</div><div class="line"><span class="o">==</span><span class="nv">1127561</span><span class="o">==</span>     <span class="k">in</span> use at exit: <span class="m">4</span>,096 bytes <span class="k">in</span> <span class="m">1</span> <span class="nv">blocks</span>
</div><div class="line"><span class="o">==</span><span class="nv">1127561</span><span class="o">==</span>   total heap usage: <span class="m">11</span> allocs, <span class="m">10</span> frees, <span class="m">6</span>,253 bytes <span class="nv">allocated</span>
</div><div class="line"><span class="o">==</span><span class="nv">1127561</span><span class="o">==</span>
</div><div class="line"><span class="o">==</span><span class="nv">1127561</span><span class="o">==</span> LEAK SUMMARY:
</div><div class="line"><span class="o">==</span><span class="nv">1127561</span><span class="o">==</span>    definitely lost: <span class="m">4</span>,096 bytes <span class="k">in</span> <span class="m">1</span> <span class="nv">blocks</span>
</div><div class="line"><span class="o">==</span><span class="nv">1127561</span><span class="o">==</span>    indirectly lost: <span class="m">0</span> bytes <span class="k">in</span> <span class="m">0</span> <span class="nv">blocks</span>
</div><div class="line"><span class="o">==</span><span class="nv">1127561</span><span class="o">==</span>      possibly lost: <span class="m">0</span> bytes <span class="k">in</span> <span class="m">0</span> <span class="nv">blocks</span>
</div><div class="line"><span class="o">==</span><span class="nv">1127561</span><span class="o">==</span>    still reachable: <span class="m">0</span> bytes <span class="k">in</span> <span class="m">0</span> <span class="nv">blocks</span>
</div><div class="line"><span class="o">==</span><span class="nv">1127561</span><span class="o">==</span>         suppressed: <span class="m">0</span> bytes <span class="k">in</span> <span class="m">0</span> <span class="nv">blocks</span>
</div><div class="line"><span class="o">==</span><span class="nv">1127561</span><span class="o">==</span> Rerun with --leak-check<span class="o">=</span>full to see details of leaked <span class="nv">memory</span>
</div><div class="line"><span class="o">==</span><span class="nv">1127561</span><span class="o">==</span>
</div><div class="line"><span class="o">==</span><span class="nv">1127561</span><span class="o">==</span> For lists of detected and suppressed errors, rerun with: -s
</div><div class="line"><span class="o">==</span><span class="nv">1127561</span><span class="o">==</span> ERROR SUMMARY: <span class="m">0</span> errors from <span class="m">0</span> contexts <span class="o">(</span>suppressed: <span class="m">0</span> from <span class="m">0</span><span class="o">)</span>
</div></code></pre></div>
</div>
<p>Want to run unit tests with Valgrind? You can do that too</p>
<div class="block-code" data-language="bash"><div class="highlight"><pre><span></span><code><div class="line">$ cargo <span class="nb">test</span>
</div><div class="line">   Compiling example v0.1.0 <span class="o">(</span>/root/code/example<span class="o">)</span>
</div><div class="line">    Finished <span class="nb">test</span> <span class="o">[</span>unoptimized + debuginfo<span class="o">]</span> target<span class="o">(</span>s<span class="o">)</span> <span class="k">in</span> <span class="m">1</span>.37s
</div><div class="line">     Running unittests src/main.rs <span class="o">(</span>target/debug/deps/example-64baa11f6262b62f<span class="o">)</span>
</div><div class="line">
</div><div class="line">running <span class="m">0</span> tests
</div><div class="line">
</div><div class="line"><span class="nb">test</span> result: ok. <span class="m">0</span> passed<span class="p">;</span> <span class="m">0</span> failed<span class="p">;</span> <span class="m">0</span> ignored<span class="p">;</span> <span class="m">0</span> measured<span class="p">;</span> <span class="m">0</span> filtered out<span class="p">;</span> finished <span class="k">in</span> <span class="m">0</span>.00s
</div><div class="line">$ valgrind target/debug/deps/example-64baa11f6262b62f
</div><div class="line"><span class="o">==</span><span class="nv">1134151</span><span class="o">==</span> Memcheck, a memory error <span class="nv">detector</span>
</div><div class="line"><span class="o">==</span><span class="nv">1134151</span><span class="o">==</span> Copyright <span class="o">(</span>C<span class="o">)</span> <span class="m">2002</span>-2022, and GNU GPL<span class="err">&#39;</span>d, by Julian Seward et al.
</div><div class="line"><span class="o">==</span><span class="nv">1134151</span><span class="o">==</span> Using Valgrind-3.19.0 and LibVEX<span class="p">;</span> rerun with -h <span class="k">for</span> copyright <span class="nv">info</span>
</div><div class="line"><span class="o">==</span><span class="nv">1134151</span><span class="o">==</span> Command: target/debug/deps/example-64baa11f6262b62f
</div><div class="line"><span class="o">==</span><span class="nv">1134151</span><span class="o">==</span>
</div><div class="line">
</div><div class="line">running <span class="m">1</span> <span class="nb">test</span>
</div><div class="line"><span class="nb">test</span> test::test_leak ... ok
</div><div class="line">
</div><div class="line"><span class="nb">test</span> result: ok. <span class="m">1</span> passed<span class="p">;</span> <span class="m">0</span> failed<span class="p">;</span> <span class="m">0</span> ignored<span class="p">;</span> <span class="m">0</span> measured<span class="p">;</span> <span class="m">0</span> filtered out<span class="p">;</span> finished <span class="k">in</span> <span class="m">0</span>.26s
</div><div class="line">
</div><div class="line"><span class="o">==</span><span class="nv">1134151</span><span class="o">==</span>
</div><div class="line"><span class="o">==</span><span class="nv">1134151</span><span class="o">==</span> HEAP SUMMARY:
</div><div class="line"><span class="o">==</span><span class="nv">1134151</span><span class="o">==</span>     <span class="k">in</span> use at exit: <span class="m">4</span>,096 bytes <span class="k">in</span> <span class="m">1</span> <span class="nv">blocks</span>
</div><div class="line"><span class="o">==</span><span class="nv">1134151</span><span class="o">==</span>   total heap usage: <span class="m">669</span> allocs, <span class="m">668</span> frees, <span class="m">72</span>,979 bytes <span class="nv">allocated</span>
</div><div class="line"><span class="o">==</span><span class="nv">1134151</span><span class="o">==</span>
</div><div class="line"><span class="o">==</span><span class="nv">1134151</span><span class="o">==</span> LEAK SUMMARY:
</div><div class="line"><span class="o">==</span><span class="nv">1134151</span><span class="o">==</span>    definitely lost: <span class="m">4</span>,096 bytes <span class="k">in</span> <span class="m">1</span> <span class="nv">blocks</span>
</div><div class="line"><span class="o">==</span><span class="nv">1134151</span><span class="o">==</span>    indirectly lost: <span class="m">0</span> bytes <span class="k">in</span> <span class="m">0</span> <span class="nv">blocks</span>
</div><div class="line"><span class="o">==</span><span class="nv">1134151</span><span class="o">==</span>      possibly lost: <span class="m">0</span> bytes <span class="k">in</span> <span class="m">0</span> <span class="nv">blocks</span>
</div><div class="line"><span class="o">==</span><span class="nv">1134151</span><span class="o">==</span>    still reachable: <span class="m">0</span> bytes <span class="k">in</span> <span class="m">0</span> <span class="nv">blocks</span>
</div><div class="line"><span class="o">==</span><span class="nv">1134151</span><span class="o">==</span>         suppressed: <span class="m">0</span> bytes <span class="k">in</span> <span class="m">0</span> <span class="nv">blocks</span>
</div><div class="line"><span class="o">==</span><span class="nv">1134151</span><span class="o">==</span> Rerun with --leak-check<span class="o">=</span>full to see details of leaked <span class="nv">memory</span>
</div><div class="line"><span class="o">==</span><span class="nv">1134151</span><span class="o">==</span>
</div><div class="line"><span class="o">==</span><span class="nv">1134151</span><span class="o">==</span> For lists of detected and suppressed errors, rerun with: -s
</div><div class="line"><span class="o">==</span><span class="nv">1134151</span><span class="o">==</span> ERROR SUMMARY: <span class="m">0</span> errors from <span class="m">0</span> contexts <span class="o">(</span>suppressed: <span class="m">0</span> from <span class="m">0</span><span class="o">)</span>
</div></code></pre></div>
</div>
<p>Use Valgrind on binary is easy, but for unit tests you need to supply the unit test binary path which can change between builds, and if you have lots of test binaries, it's tedious to find the path and feed it to Valgrind by hand.</p>
<h2>Use <code>cargo-valgrind</code></h2>
<p><a href="https://github.com/jfrimmel/cargo-valgrind">cargo-valgrind</a> is a cargo subcommand that extends cargo with the capability to directly run Valgrind on any crate executable, it makes running <code>cargo test</code> with Valgrind quite easy.</p>
<div class="block-code" data-language="bash"><div class="highlight"><pre><span></span><code><div class="line">$ cargo valgrind run
</div><div class="line">    Finished dev <span class="o">[</span>unoptimized + debuginfo<span class="o">]</span> target<span class="o">(</span>s<span class="o">)</span> <span class="k">in</span> <span class="m">0</span>.02s
</div><div class="line">     Running <span class="sb">`</span>/root/.cargo/bin/cargo-valgrind target/debug/example<span class="sb">`</span>
</div><div class="line">       Error leaked <span class="m">4</span>.0 kiB <span class="k">in</span> <span class="m">1</span> block
</div><div class="line">        Info at calloc <span class="o">(</span>vg_replace_malloc.c:1328<span class="o">)</span>
</div><div class="line">             at alloc::alloc::alloc_zeroed <span class="o">(</span>alloc.rs:160<span class="o">)</span>
</div><div class="line">             at alloc::alloc::Global::alloc_impl <span class="o">(</span>alloc.rs:171<span class="o">)</span>
</div><div class="line">             at &lt;alloc::alloc::Global as core::alloc::Allocator&gt;::allocate_zeroed <span class="o">(</span>alloc.rs:236<span class="o">)</span>
</div><div class="line">             at alloc::raw_vec::RawVec&lt;T,A&gt;::allocate_in <span class="o">(</span>raw_vec.rs:186<span class="o">)</span>
</div><div class="line">             at alloc::raw_vec::RawVec&lt;T,A&gt;::with_capacity_zeroed_in <span class="o">(</span>raw_vec.rs:139<span class="o">)</span>
</div><div class="line">             at &lt;T as alloc::vec::spec_from_elem::SpecFromElem&gt;::from_elem <span class="o">(</span>spec_from_elem.rs:54<span class="o">)</span>
</div><div class="line">             at alloc::vec::from_elem <span class="o">(</span>mod.rs:2454<span class="o">)</span>
</div><div class="line">             at example::leak <span class="o">(</span>main.rs:2<span class="o">)</span>
</div><div class="line">             at example::main <span class="o">(</span>main.rs:7<span class="o">)</span>
</div><div class="line">             at core::ops::function::FnOnce::call_once <span class="o">(</span><span class="k">function</span>.rs:248<span class="o">)</span>
</div><div class="line">             at std::sys_common::backtrace::__rust_begin_short_backtrace <span class="o">(</span>backtrace.rs:122<span class="o">)</span>
</div><div class="line">     Summary Leaked <span class="m">4</span>.0 kiB total
</div><div class="line">$ cargo valgrind <span class="nb">test</span>
</div><div class="line">    Finished <span class="nb">test</span> <span class="o">[</span>unoptimized + debuginfo<span class="o">]</span> target<span class="o">(</span>s<span class="o">)</span> <span class="k">in</span> <span class="m">0</span>.02s
</div><div class="line">     Running unittests src/main.rs <span class="o">(</span>target/debug/deps/example-64baa11f6262b62f<span class="o">)</span>
</div><div class="line">
</div><div class="line">running <span class="m">1</span> <span class="nb">test</span>
</div><div class="line"><span class="nb">test</span> test::test_leak ... ok
</div><div class="line">
</div><div class="line"><span class="nb">test</span> result: ok. <span class="m">1</span> passed<span class="p">;</span> <span class="m">0</span> failed<span class="p">;</span> <span class="m">0</span> ignored<span class="p">;</span> <span class="m">0</span> measured<span class="p">;</span> <span class="m">0</span> filtered out<span class="p">;</span> finished <span class="k">in</span> <span class="m">0</span>.26s
</div><div class="line">
</div><div class="line">       Error leaked <span class="m">4</span>.0 kiB <span class="k">in</span> <span class="m">1</span> block
</div><div class="line">        Info at calloc <span class="o">(</span>vg_replace_malloc.c:1328<span class="o">)</span>
</div><div class="line">             at alloc::alloc::alloc_zeroed <span class="o">(</span>alloc.rs:160<span class="o">)</span>
</div><div class="line">             at alloc::alloc::Global::alloc_impl <span class="o">(</span>alloc.rs:171<span class="o">)</span>
</div><div class="line">             at &lt;alloc::alloc::Global as core::alloc::Allocator&gt;::allocate_zeroed <span class="o">(</span>alloc.rs:236<span class="o">)</span>
</div><div class="line">             at alloc::raw_vec::RawVec&lt;T,A&gt;::allocate_in <span class="o">(</span>raw_vec.rs:186<span class="o">)</span>
</div><div class="line">             at alloc::raw_vec::RawVec&lt;T,A&gt;::with_capacity_zeroed_in <span class="o">(</span>raw_vec.rs:139<span class="o">)</span>
</div><div class="line">             at &lt;T as alloc::vec::spec_from_elem::SpecFromElem&gt;::from_elem <span class="o">(</span>spec_from_elem.rs:54<span class="o">)</span>
</div><div class="line">             at alloc::vec::from_elem <span class="o">(</span>mod.rs:2454<span class="o">)</span>
</div><div class="line">             at example::leak <span class="o">(</span>main.rs:2<span class="o">)</span>
</div><div class="line">             at example::test::test_leak <span class="o">(</span>main.rs:16<span class="o">)</span>
</div><div class="line">             at example::test::test_leak::<span class="o">{{</span>closure<span class="o">}}</span> <span class="o">(</span>main.rs:15<span class="o">)</span>
</div><div class="line">             at core::ops::function::FnOnce::call_once <span class="o">(</span><span class="k">function</span>.rs:248<span class="o">)</span>
</div><div class="line">     Summary Leaked <span class="m">4</span>.0 kiB total
</div><div class="line">error: <span class="nb">test</span> failed, to rerun pass <span class="s1">&#39;--bin example&#39;</span>
</div></code></pre></div>
</div>
<p>cargo-valgrind has a nice colored output and will fail the test if memory errors detected, but as of v2.1.0 it can not output just raw Valgrind output.</p>
<h2>Use Cargo runner</h2>
<p>Cargo also supports setting a <a href="https://doc.rust-lang.org/cargo/reference/config.html#targettriplerunner">custom target runner</a> which is useful for cross testing, for example, you can cross compile a Rust program from Linux to Windows target and use <code>wine</code> to run unit tests.</p>
<p>We can also leverage this feature to run unit tests under Valgrind:</p>
<div class="block-code" data-language="bash"><div class="highlight"><pre><span></span><code><div class="line">$ <span class="nv">CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUNNER</span><span class="o">=</span><span class="s2">&quot;valgrind --error-exitcode=1&quot;</span> cargo <span class="nb">test</span>
</div><div class="line">    Finished <span class="nb">test</span> <span class="o">[</span>unoptimized + debuginfo<span class="o">]</span> target<span class="o">(</span>s<span class="o">)</span> <span class="k">in</span> <span class="m">0</span>.03s
</div><div class="line">     Running unittests src/main.rs <span class="o">(</span>target/debug/deps/example-64baa11f6262b62f<span class="o">)</span>
</div><div class="line"><span class="o">==</span><span class="nv">1136560</span><span class="o">==</span> Memcheck, a memory error <span class="nv">detector</span>
</div><div class="line"><span class="o">==</span><span class="nv">1136560</span><span class="o">==</span> Copyright <span class="o">(</span>C<span class="o">)</span> <span class="m">2002</span>-2022, and GNU GPL<span class="err">&#39;</span>d, by Julian Seward et al.
</div><div class="line"><span class="o">==</span><span class="nv">1136560</span><span class="o">==</span> Using Valgrind-3.19.0 and LibVEX<span class="p">;</span> rerun with -h <span class="k">for</span> copyright <span class="nv">info</span>
</div><div class="line"><span class="o">==</span><span class="nv">1136560</span><span class="o">==</span> Command: /root/code/example/target/debug/deps/example-64baa11f6262b62f
</div><div class="line"><span class="o">==</span><span class="nv">1136560</span><span class="o">==</span>
</div><div class="line">
</div><div class="line">running <span class="m">1</span> <span class="nb">test</span>
</div><div class="line"><span class="nb">test</span> test::test_leak ... ok
</div><div class="line">
</div><div class="line"><span class="nb">test</span> result: ok. <span class="m">1</span> passed<span class="p">;</span> <span class="m">0</span> failed<span class="p">;</span> <span class="m">0</span> ignored<span class="p">;</span> <span class="m">0</span> measured<span class="p">;</span> <span class="m">0</span> filtered out<span class="p">;</span> finished <span class="k">in</span> <span class="m">0</span>.27s
</div><div class="line">
</div><div class="line"><span class="o">==</span><span class="nv">1136560</span><span class="o">==</span>
</div><div class="line"><span class="o">==</span><span class="nv">1136560</span><span class="o">==</span> HEAP SUMMARY:
</div><div class="line"><span class="o">==</span><span class="nv">1136560</span><span class="o">==</span>     <span class="k">in</span> use at exit: <span class="m">4</span>,096 bytes <span class="k">in</span> <span class="m">1</span> <span class="nv">blocks</span>
</div><div class="line"><span class="o">==</span><span class="nv">1136560</span><span class="o">==</span>   total heap usage: <span class="m">669</span> allocs, <span class="m">668</span> frees, <span class="m">72</span>,998 bytes <span class="nv">allocated</span>
</div><div class="line"><span class="o">==</span><span class="nv">1136560</span><span class="o">==</span>
</div><div class="line"><span class="o">==</span><span class="nv">1136560</span><span class="o">==</span> LEAK SUMMARY:
</div><div class="line"><span class="o">==</span><span class="nv">1136560</span><span class="o">==</span>    definitely lost: <span class="m">4</span>,096 bytes <span class="k">in</span> <span class="m">1</span> <span class="nv">blocks</span>
</div><div class="line"><span class="o">==</span><span class="nv">1136560</span><span class="o">==</span>    indirectly lost: <span class="m">0</span> bytes <span class="k">in</span> <span class="m">0</span> <span class="nv">blocks</span>
</div><div class="line"><span class="o">==</span><span class="nv">1136560</span><span class="o">==</span>      possibly lost: <span class="m">0</span> bytes <span class="k">in</span> <span class="m">0</span> <span class="nv">blocks</span>
</div><div class="line"><span class="o">==</span><span class="nv">1136560</span><span class="o">==</span>    still reachable: <span class="m">0</span> bytes <span class="k">in</span> <span class="m">0</span> <span class="nv">blocks</span>
</div><div class="line"><span class="o">==</span><span class="nv">1136560</span><span class="o">==</span>         suppressed: <span class="m">0</span> bytes <span class="k">in</span> <span class="m">0</span> <span class="nv">blocks</span>
</div><div class="line"><span class="o">==</span><span class="nv">1136560</span><span class="o">==</span> Rerun with --leak-check<span class="o">=</span>full to see details of leaked <span class="nv">memory</span>
</div><div class="line"><span class="o">==</span><span class="nv">1136560</span><span class="o">==</span>
</div><div class="line"><span class="o">==</span><span class="nv">1136560</span><span class="o">==</span> For lists of detected and suppressed errors, rerun with: -s
</div><div class="line"><span class="o">==</span><span class="nv">1136560</span><span class="o">==</span> ERROR SUMMARY: <span class="m">0</span> errors from <span class="m">0</span> contexts <span class="o">(</span>suppressed: <span class="m">0</span> from <span class="m">0</span><span class="o">)</span>
</div></code></pre></div>
</div>
<hr /><p><a rel="payment" href="https://afdian.com/a/messense">爱发电上赞助</a></p>]]></content:encoded></item><item><title><![CDATA[aliyundrive-webdav One Year Later]]></title><guid>https://messense.me/aliyundrive-webdav-one-year-later</guid><link>https://messense.me/aliyundrive-webdav-one-year-later</link><pubDate>Sun, 14 Aug 2022 07:30:45 +0000</pubDate><content:encoded><![CDATA[<p>Time flies, 我的阿里云盘 WebDAV 项目马上就满一年了。</p>
<div class="photo"><figure><picture><source srcset="https://i.typlog.com/messense/8339538121_884378.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_800/format,webp 1x, https://i.typlog.com/messense/8339538121_884378.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_1600/format,webp 2x" media="(min-width: 800px)" type="image/webp"><source srcset="https://i.typlog.com/messense/8339538121_884378.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_800 1x, https://i.typlog.com/messense/8339538121_884378.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_1600 2x" media="(min-width: 800px)"><img src="https://i.typlog.com/messense/8339538121_884378.png" alt="WX20220814-134741@2x.png"loading="lazy" decoding="async" width="1834" height="718" /></picture></figure></div><h2>功能迭代</h2>
<p>过去一年，断断续续地开发和维护，增加了一些新功能，这里简单盘点一下。</p>
<h3>1、阿里云 PDS 支持</h3>
<p><a href="https://www.aliyun.com/product/storage/pds">阿里云相册与网盘服务（PDS）</a>是阿里云盘使用的底层能力，在阿里云上公开售卖，其 API 接口和阿里云盘的接口基本类似，支持起来比较简单，命令行传入 <code>--domain-id</code> 参数即可使用。如果有自建网盘的需求，使用这个功能可以获得接近阿里云盘的体验。</p>
<h3>2、只读模式</h3>
<p>WebDAV 协议支持读写，有用户反馈配置给家里长辈观看时，不知怎的把文件夹都删除了。通过增加只读模式禁止写操作，避免出现这里情况。命令行传入 <code>--read-only</code> 参数即可开启。</p>
<h3>3、HTTPS 支持</h3>
<p>基于 <a href="https://github.com/rustls/rustls">rustls</a> 开发了 TLS/HTTPS 服务端支持，低端设备如路由器上可能不会专门部署一个 Nginx 来做 TLS termination，内置 HTTPS 支持方便使用。</p>
<p>目前还没有做自动获取比如 Let's Encrypt 的 HTTPS 证书功能，可以用 certbot 或者 acme.sh 等工具签发证书后使用。</p>
<p>未来一个可能有点意思的扩展是通过 <a href="https://tailscale.com">Tailscale</a> 获得私有的 <a href="https://tailscale.com/kb/1136/tailnet/">tailnet</a> 的 HTTPS 证书，将服务部署在 tailnet 中随时随地远程安全访问。</p>
<h3>4、iOS Live Photo 下载</h3>
<p>阿里云盘的接口没有提供原始的 iOS Live Photo （<code>.livp</code> 格式）下载地址，导致尝试下载此类文件时 <code>500</code> 错误。这个问题可能是 PDS 或者阿里云盘的锅，很难理解为什么不提供 <code>.livp</code> 的下载地址，其本质上只是个包含一个视频和多个图片的 <code>zip</code> 压缩包。</p>
<div class="photo"><figure><picture><source srcset="https://i.typlog.com/messense/8339538106_050296.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_259/format,webp" type="image/webp"><img src="https://i.typlog.com/messense/8339538106_050296.png" alt="image.png"loading="lazy" decoding="async" width="259" height="194" /></picture></figure></div><p>通过识别下载请求的文件格式为 <code>.livp</code> 然后实时下载对应的视频和图片文件，并将他们组合生成为一个新的 <code>.livp</code> 文件实现了 <code>.livp</code> 格式文件的下载的功能，不过缺点就是会丢失文件创建/修改时间的信息，对于文件同步的场景就很不友好。</p>
<h3>5、扫码登录</h3>
<p>一直以来都需要从阿里云盘 Web 端获取 <code>refresh token</code> 来完成登录授权，整个流程对小白用户来说还是过于复杂。</p>
<p><code>1.4.0</code> 版本 <a href="https://github.com/gngpp">@gngpp</a> 贡献了命令行扫码登录功能，可以直接使用阿里云盘移动端 App 扫描二维码登录并自动获取 <code>refresh token</code>。</p>
<div class="photo"><figure><picture><source srcset="https://i.typlog.com/messense/8339538085_454258.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_800/format,webp 1x, https://i.typlog.com/messense/8339538085_454258.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_1600/format,webp 2x" media="(min-width: 800px)" type="image/webp"><source srcset="https://i.typlog.com/messense/8339538085_454258.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_800 1x, https://i.typlog.com/messense/8339538085_454258.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_1600 2x" media="(min-width: 800px)"><img src="https://i.typlog.com/messense/8339538085_454258.png" alt="image.png"loading="lazy" decoding="async" width="1702" height="1394" /></picture></figure></div><p>而后在 <code>1.7.1</code> 版本中，给 OpenWrt 配置后台也增加了扫码登录功能，扫码后自动填充 <code>refresh token</code> 到输入框中，非常便捷</p>
<div class="photo"><figure><picture><source srcset="https://i.typlog.com/messense/8339538068_083968.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_1396/format,webp" type="image/webp"><img src="https://i.typlog.com/messense/8339538068_083968.png" alt="image.png"loading="lazy" decoding="async" width="1396" height="972" /></picture></figure></div><h3>6、Checksums</h3>
<p>WebDAV 协议一直被诟病的缺点之一是不支持 checksums，这意味着用户无法验证上传/下载文件的完整性，<a href="https://owncloud.com">owncloud</a> 增加了私有的 <code>oc:checksums</code> 属性扩展实现了文件完整性校验，这个扩展后来也被 <a href="https://nextcloud.com/">Nextcloud</a> 和 <a href="https://rclone.org/">rclone</a> 支持。</p>
<p><code>1.8.4</code> 版本也增加了对 <code>oc:checksums</code> 扩展的支持，从而在通过 rclone 挂载时可以校验文件完整性。由于阿里云盘接口只返回了 sha1，目前推荐使用 rclone 的 Nextcloud WebDAV 模式，不仅可以校验下载文件的完整性，也可以自动跳过上传相同 sha1 值的文件。</p>
<h2>项目洞察</h2>
<div class="photo"><figure><picture><source srcset="https://i.typlog.com/messense/8339538053_588649.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_1474/format,webp" type="image/webp"><img src="https://i.typlog.com/messense/8339538053_588649.png" alt="image.png"loading="lazy" decoding="async" width="1474" height="1010" /></picture></figure></div><p>过去一年项目在 GitHub 上获得了 4900+ 个 stars，稳定增长。</p>
<div class="photo"><figure><picture><source srcset="https://i.typlog.com/messense/8339538036_128664.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_980/format,webp" type="image/webp"><img src="https://i.typlog.com/messense/8339538036_128664.png" alt="image.png"loading="lazy" decoding="async" width="980" height="620" /></picture></figure></div><p>总共有 11 个代码贡献者，相对来说还是比较少的，Rust 语言的门槛相对较高。GitHub 上产生了 217 个 issues 和约 190 个讨论区主题。</p>
<div class="photo"><figure><picture><source srcset="https://i.typlog.com/messense/8339538023_382544.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_800/format,webp 1x, https://i.typlog.com/messense/8339538023_382544.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_1600/format,webp 2x" media="(min-width: 800px)" type="image/webp"><source srcset="https://i.typlog.com/messense/8339538023_382544.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_800 1x, https://i.typlog.com/messense/8339538023_382544.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_1600 2x" media="(min-width: 800px)"><img src="https://i.typlog.com/messense/8339538023_382544.png" alt="image.png"loading="lazy" decoding="async" width="1802" height="800" /></picture></figure></div><p>GitHub 仓库每天 PV 约 1500, UV 约 600.</p>
<div class="photo"><figure><picture><source srcset="https://i.typlog.com/messense/8339538010_414964.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_800/format,webp 1x, https://i.typlog.com/messense/8339538010_414964.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_1600/format,webp 2x" media="(min-width: 800px)" type="image/webp"><source srcset="https://i.typlog.com/messense/8339538010_414964.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_800 1x, https://i.typlog.com/messense/8339538010_414964.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_1600 2x" media="(min-width: 800px)"><img src="https://i.typlog.com/messense/8339538010_414964.png" alt="image.png"loading="lazy" decoding="async" width="1658" height="760" /></picture></figure></div><p>B 站和 YouTube 上的一些视频教程带来了不少流量，比如</p>
<iframe src="//player.bilibili.com/player.html?aid=593039221&bvid=BV1Jq4y1y7Xb&cid=483119725&page=1" width="100%" height="500" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

<iframe src="//player.bilibili.com/player.html?aid=633955874&bvid=BV1Fb4y1875B&cid=434341379&page=1" width="100%" height="500" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

<h2>Fun Facts</h2>
<h4>为什么选择 Rust 语言？</h4>
<p>TLDR: I like it.</p>
<p>其实 Rust 语言标榜的内存安全、零开销抽象、fearless concurrency 等特性在这个客户端项目上并不是特别重要，你用 Java、Go、Python 来写也没毛病，实际上也确实有很多其他语言版本的实现。</p>
<p>Rust 对主流平台、CPU 架构的优秀的支持是这个项目流行的关键。除了常见的 x86、x86_64 架构，还支持 arm64、armv7、armv6、armv5、mips 和 mipsel 等 CPU 架构，特别是 mips 和 arm 架构在路由器上使用的很多。</p>
<h4>能不能赚钱？</h4>
<p>显然不能。</p>
<p>作为一个 side project，主要还是满足自己的使用需求。有时候解决一些用户的问题/需求，会收获一些打赏，大部分人都是白嫖🙄。项目流行起来以后，也有了个广告合作，够一个月两杯咖啡罢了。</p>
<hr /><p><a rel="payment" href="https://afdian.com/a/messense">爱发电上赞助</a></p>]]></content:encoded></item><item><title><![CDATA[基于 Jellyfin 和 aliyundrive-fuse 打造家庭媒体库]]></title><guid>https://messense.me/aliyundrive-fuse</guid><link>https://messense.me/aliyundrive-fuse</link><pubDate>Sat, 22 Jan 2022 10:29:22 +0000</pubDate><content:encoded><![CDATA[<h2>前言</h2>
<p><a href="https://messense.me/aliyundrive-webdav">之前介绍过</a>的 <a href="https://github.com/messense/aliyundrive-dav">aliyundrive-webdav</a> 项目项目已经实现了通过 WebDAV 访问阿里云盘内容，在其基础上配合 <a href="https://rclone.org">rclone</a> 已经可以实现将 WebDAV 挂载为本地磁盘后提供给 <a href="https://emby.media/">Emby</a>/<a href="https://jellyfin.org/">Jellyfin</a> 之类的多媒体内容管理系统使用。虽然这样基本能够工作，但过于复杂了，增加 rclone 只是因为 Jellyfin 之类的软件原生不支持直连 WebDAV 服务，所以为了去掉中间商 rclone，一种方法是给 Jellyfin 写个支持读取 WebDAV 的插件，而这需要使用 C# 语言实现并且需要学习了解 Jellyfin 的插件接口，不是很熟故放弃这条路径；二是直接实现与 rclone 类似的挂载为本地磁盘的功能，Linux 和 macOS 上都支持 FUSE(Filesystem in Userspace)，Rust 也有对应的库 <a href="https://crates.io/crates/fuser">fuser</a> 可以直接使用，于是便开发了 <a href="https://github.com/messense/aliyundrive-fuse">aliyundrive-fuse</a> 这个项目。</p>
<h2>aliyundrive-fuse</h2>
<p>之前 aliyundrive-webdav 的已有代码是面向 Rust 异步 async/await 特性设计的，而目前 fuser 还不支持异步只有同步的接口，故将之前访问阿里云盘的接口从异步改成了同步实现，得益于 <a href="https://crates.io/crates/reqwest">reqwest</a> 库同时提供了基本一致的异步和同步接口，<a href="https://github.com/messense/aliyundrive-fuse/commit/9b8c9e0fbba4277b41dfefa4aa40d054e9c1fe75">改造成本很低</a>。而后只需要实现 fuser 的 <a href="https://docs.rs/fuser/latest/fuser/trait.Filesystem.html">Filesystem</a> trait 的几个方法就可以实现一个简单的只读文件系统。</p>
<p>相对比较麻烦的地方是，由于 FUSE 文件系统单次读请求的大小都比较小，比如 Linux 上最大也就 128KB，这对于需要发起网络请求下载数据的场景就是灾难，因此需要缓存一个相对较大的大小比如 10MB，来避免每次读取都需要发起网络请求。解决了这个问题，文件读取速度就有了较好的保障，但因为缓存一次要下载 10MB 的数据，在读取到还没有被缓存的位置的时候需要等待网络请求传输完成，结果就可能导致视频在串流播放的时候偶尔出现卡顿的情况。不过只要播放器也会提前下载后续需要播放的内容倒也不是大问题，至少在配合 Jellyfin 使用的情况下还是挺流畅的没有明显问题。</p>
<h2>安装配置</h2>
<p>aliyundrive-fuse 提供了<a href="https://github.com/messense/aliyundrive-fuse#%E5%AE%89%E8%A3%85">很多种安装方式</a>，最简单的可以通过 pip 安装：<code>pip install aliyundrive-fuse</code> ，安装好之后提前通过阿里云盘网页版本获取 <code>refresh token</code>，然后就可以配置挂载了，以 Linux 系统为例，这里我们在 <code>/mnt/aliyundrive</code> 目录挂载磁盘：</p>
<div class="block-code" data-language="bash"><div class="highlight"><pre><span></span><code><div class="line"><span class="c1"># 先新建目录</span>
</div><div class="line">mkdir -p /mnt/aliyundrive /etc/aliyundrive-fuse
</div><div class="line"><span class="c1"># 挂载磁盘</span>
</div><div class="line">aliyundrive-fuse -r your-refresh-token -w /etc/aliyundrive-fuse /mnt/aliyundrive
</div></code></pre></div>
</div>
<p>其中 <code>your-refresh-token</code> 替换为你的云盘 <code>refresh token</code> 值。</p>
<p>以 Docker 安装的 Jellyfin 为例，我们需要将刚刚挂载好的阿里云盘目录也 mount 到 Docker 容器中，这里假设 Jellyfin 工作路径为 <code>/root/jellyfin</code> 将云盘挂载到容器 <code>/media</code> 路径：</p>
<div class="block-code" data-language="bash"><div class="highlight"><pre><span></span><code><div class="line">docker run -d --name jellyfin <span class="se">\</span>
</div><div class="line">  -v /root/jellyfin/config:/config <span class="se">\</span>
</div><div class="line">  -v /root/jellyfin/cache:/cache <span class="se">\</span>
</div><div class="line">  -v /mnt/aliyundrive:/media <span class="se">\</span>
</div><div class="line">  -p <span class="m">8096</span>:8096 <span class="se">\</span>
</div><div class="line">  --device<span class="o">=</span>/dev/dri/renderD128 <span class="se">\</span>
</div><div class="line">  --device /dev/dri/card0:/dev/dri/card0 <span class="se">\</span>
</div><div class="line">  --restart unless-stopped <span class="se">\</span>
</div><div class="line">  jellyfin/jellyfin
</div></code></pre></div>
</div>
<p>然后访问 Jellyfin 容器的 8096 端口进入控制台添加媒体库。</p>
<h2>媒体库管理</h2>
<p>在媒体库设置中将文件夹的位置选择挂载好的阿里网盘即可：
<img src="https://i.typlog.com/messense/8357153280_331442.png" alt="image.png" /></p>
<p>等待媒体库扫描完成后即可</p>
<div class="gallery"><div class="gallery_column"><figure><picture><source srcset="https://i.typlog.com/messense/8357152791_127789.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_800/format,webp 1x, https://i.typlog.com/messense/8357152791_127789.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_1600/format,webp 2x" media="(min-width: 800px)" type="image/webp"><source srcset="https://i.typlog.com/messense/8357152791_127789.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_800 1x, https://i.typlog.com/messense/8357152791_127789.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_1600 2x" media="(min-width: 800px)"><img src="https://i.typlog.com/messense/8357152791_127789.png" alt="jellyfin-cover.png"loading="lazy" decoding="async" width="2864" height="1354" /></picture></figure></div><div class="gallery_column"><figure><picture><source srcset="https://i.typlog.com/messense/8357152783_05519.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_800/format,webp 1x, https://i.typlog.com/messense/8357152783_05519.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_1600/format,webp 2x" media="(min-width: 800px)" type="image/webp"><source srcset="https://i.typlog.com/messense/8357152783_05519.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_800 1x, https://i.typlog.com/messense/8357152783_05519.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_1600 2x" media="(min-width: 800px)"><img src="https://i.typlog.com/messense/8357152783_05519.png" alt="jellyfin-play.png"loading="lazy" decoding="async" width="2864" height="1350" /></picture></figure></div></div>
<p>Enjoy!</p>
<hr /><p><a rel="payment" href="https://afdian.com/a/messense">爱发电上赞助</a></p>]]></content:encoded></item><item><title><![CDATA[aliyundrive-webdav 诞生记]]></title><guid>https://messense.me/aliyundrive-webdav</guid><link>https://messense.me/aliyundrive-webdav</link><dc:creator><![CDATA[messense]]></dc:creator><pubDate>Tue, 16 Nov 2021 15:28:56 +0000</pubDate><content:encoded><![CDATA[<h2>起源</h2>
<p>2021年8月份的时候开始使用阿里云盘，某天在 V2EX 上看了一个帖子<a href="https://v2ex.com/t/796188">《当 [Infuse] 遇上 [阿里云盘] 🚀》</a>了解到可以通过 <a href="https://github.com/zxbu">@zxbu</a> 开源的<a href="https://github.com/zxbu/webdav-aliyundriver">webdav-aliyundriver</a> 项目给阿里云盘添加 WebDAV 协议支持，从而可以将阿里云盘通过 rclone、davfs 和 RaiDrive 等软件挂载为磁盘使用，或者通过 Infuse 访问作为媒体服务器，而 Infuse 支持 Apple TV 并且能够自动扫描媒体文件抓取封面、字幕等构建媒体库，作为一个 Apple TV 用户自然很喜欢看云盘视频内容更方便了。</p>
<p>前不久也入手了一个 <a href="http://wiki.friendlyarm.com/wiki/index.php/NanoPi_R4S/zh">NanoPi R4S</a> 软路由，配置一个双核 Cortex-A72 和一个四核 Cortex-A53、4G 内存性能不错可以跑 Docker，想着可以把 <a href="https://github.com/zxbu/webdav-aliyundriver">webdav-aliyundriver</a> 容器跑在软路由上，尝试了之后遇到 <code>network_mode</code> 为 <code>bridge</code> 运行报错的问题，改成 <code>host</code> 后正常。不过由于它是 Java 语言实现的，占用内存随随便便就几百兆了比较高，看着挺不舒服的，况且程序员嘛本来就喜欢自己捣鼓，于是就萌发了使用 Rust 语言自行实现一套的想法，这就诞生了 <a href="https://github.com/messense/aliyundrive-webdav">aliyundrive-webdav</a> 项目。</p>
<h2>发展</h2>
<h3>第一个版本</h3>
<p>说干就干，Chrome 网络调试分析了一遍阿里云盘网页版本的接口之后，大约花了三个晚上的时间，发布了只支持下载功能的第一个 <a href="https://github.com/messense/aliyundrive-webdav/tree/v0.1.0">0.1.0 版本</a>，从一开始就有了比较完善的基于 GitHub Actions 的 CI/CD 流程自动构建基于 musl libc 的二进制文件打包发布到 GitHub Releases，得益于 [maturin] 项目非常方便地将这个 Rust 编译的二进制文件像一个普通 Python wheel 一样发布到 PyPI 上使得主流操作系统如 macOS、Linux 和 Windows 可以通过 <code>pip install aliyundrive-webdav</code> 安装使用。</p>
<h3>OpenWrt 插件</h3>
<p>前面提到的二进制包已经可以在 NanoPi R4S 软路由上运行了，但因为没有 init 配置，在路由器重启后不能自动启动，另外程序运行需要配置阿里云盘 <code>refresh_token</code> 参数，而该参数存在有效期虽然程序会自动定时刷新，但当需要提供全新的 <code>refresh_token</code> 参数时没有配置界面不是很方便，于是便学习了 <a href="https://openwrt.org/packages/start">OpenWrt Packages 文档</a>，又在参考了一些现成的 OpenWrt 插件源代码之后完成了<a href="https://github.com/messense/aliyundrive-webdav/commit/6f8e586ce5217e5aab556db46b5efa71b1cf8821">第一个版本的 OpenWrt 插件</a>开发，同样集成了完善的 <code>.ipk</code> 文件 CI 自动编译打包流程。</p>
<p>Rust 编译的基于 musl libc 的二进制文件因为是全静态链接文件大小比较大，而很多路由器可用的存储空间都比较小，为了尽可能减少 <code>.ipk</code> 文件大小，增加了 Link Time Optimization 和 <code>opt-level = &quot;z&quot;</code>（优化产出物大小），并自动 strip debug symbols，即便如此二进制文件仍然有 5MB 左右，祭出大杀器 <a href="https://github.com/upx/upx">upx</a> 对二进制进行压缩后直接减少约 44% 至 2.2MB 左右，非常厉害。不过后来也遇到过 <a href="https://github.com/upx/upx/issues/504">upx 3.96 压缩的 MIPS 架构二进制文件启动即卡死</a>的问题，只能降级到 3.95 版本解决。</p>
<div class="photo"><figure><picture><source srcset="https://i.typlog.com/messense/8362923771_575221.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_800/format,webp 1x, https://i.typlog.com/messense/8362923771_575221.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_1600/format,webp 2x" media="(min-width: 800px)" type="image/webp"><source srcset="https://i.typlog.com/messense/8362923771_575221.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_800 1x, https://i.typlog.com/messense/8362923771_575221.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_1600 2x" media="(min-width: 800px)"><img src="https://i.typlog.com/messense/8362923771_575221.png" alt="image.png"loading="lazy" decoding="async" width="2486" height="1610" /></picture></figure></div><h3>梅林插件</h3>
<p>当时手上还有个洋垃圾网件 R8000 路由器，刷了 Koolshare 的梅林系统有个软件中心可以安装第三方插件，就想着能不能我也写个梅林插件做个简单的配置界面方便使用，在参阅了<a href="https://www.zybuluo.com/9168131/note/994332">《梅林软件中心插件开发教程详解》</a>和一些梅林插件源代码之后，做了第一个支持比较老的<a href="https://github.com/messense/aliyundrive-webdav/commit/7b89e4395fa614db20d883bb8c95cc8e3cfeaac5">梅林 ARM380 固件的 aliyudrive-webdav 插件</a>。可惜后来这个因为空调漏水，这个洋垃圾路由器壮烈牺牲。现在换了国产辣鸡红米 AC2100 纯当 AP 使用，因为是 MIPS 架构的 CPU，偶尔也能用来测试一下。</p>
<div class="photo"><figure><picture><source srcset="https://i.typlog.com/messense/8362923777_499886.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_1532/format,webp" type="image/webp"><img src="https://i.typlog.com/messense/8362923777_499886.png" alt="image.png"loading="lazy" decoding="async" width="1532" height="956" /></picture></figure></div><h3>论坛宣传</h3>
<p>迭代了几个小版本后，在 V2EX 上发帖<a href="https://v2ex.com/t/799811#reply15">《阿里云盘 WebDAV 路由器版本》</a>宣传了一下，得到了一些网友们的回应，带来了一些流量，GitHub issue 上渐渐多了一些提问和 feature requests。逐渐增加了 armv7、armv5te、x86_64 和 MIPS 等架构支持并发布了支持多种 CPU 架构的 Docker 镜像，还开发了呼声较高的上传文件功能。</p>
<h3>社区贡献</h3>
<p><a href="https://github.com/flyhigherpi">@flyhigherpi</a> 贡献了梅林 384/386 固件版本的支持，也吸引了国内 OpenWrt 社区比较活跃的 <a href="https://github.com/Beginner-Go">@Beginner-Go</a> 的贡献改进了我的菜鸡 OpenWrt package 代码使之更加标准和规范化，后来还将 aliyundrive-webdav 提交收录至国内比较流行的 <a href="https://github.com/coolsnowwolf/lede">lede</a> Lean's OpenWrt source 代码仓库，逐渐被各个 OpenWrt 固件发行版本收录编译方便用户安装使用。</p>
<p>为了支持 armv5te 和 MIPS 架构，也对依赖的两个不兼容的 Rust crates 提了 Pull Request 或者 Issue，通过与上游合作，增加了此类 32 位系统的支持，比如 <a href="https://github.com/metrics-rs/quanta/pull/55">quanta</a> 和 <a href="https://github.com/moka-rs/moka/pull/39">moka</a>。也许 aliyundrive-webdav 这个项目不会很长久，但开发过程中产出的对 Rust 社区 crates 的改进会长久留存下去，Rust 这个比较年轻的语言的生态会越来越好。</p>
<h2>Star History</h2>
<p>aliyundrive-webdav 项目至今三个月，GitHub Star 增长速度可能是我目前的开源项目中最快的，如下图</p>
<div class="photo"><figure><picture><source srcset="https://i.typlog.com/messense/8362923788_881143.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_800/format,webp 1x, https://i.typlog.com/messense/8362923788_881143.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_1600/format,webp 2x" media="(min-width: 800px)" type="image/webp"><source srcset="https://i.typlog.com/messense/8362923788_881143.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_800 1x, https://i.typlog.com/messense/8362923788_881143.png?x-oss-process=image/auto-orient,1/interlace,1/quality,q_90/resize,m_lfit,w_1600 2x" media="(min-width: 800px)"><img src="https://i.typlog.com/messense/8362923788_881143.png" alt="image.png"loading="lazy" decoding="async" width="1664" height="1100" /></picture></figure></div><h2>Takeaways</h2>
<p>这个用爱发电的开源项目，几乎没有带来任何收入，但却能让人看到开源社区有趣的一面，尽管大家互不相识，却能热情、高效地解决遇到的问题和困难。</p>
<p>另外，永远不要假设你的产品用户一定会怎么用，尽管我最初是想在 NanoPi R4S 这样的相对较高性能的软路由上跑来看视频的，也有些喜欢捡垃圾的用户却要跑在斐讯N1盒子、玩客云、威联通 NAS 等各式各样的平台上，用 rclone、davfs 或者 RaiDrive 配合 NAS 的文件同步服务比如群晖 CloudSync 单向/双向同步文件。</p>
<p>因为我目前工作在蚂蚁集团，正好也在阿里云盘内部反馈钉钉群中，也看到一些内部用户向官方团队提出需要 TV 版本或者 WebDAV 协议支持的需求，一开始他们是拒绝的回复说 WebDAV 是很老的协议兼容性也不好什么的，后来有一天改口说 WebDAV 协议支持设计中了，期待官方支持上线的那一天。</p>
<p>[maturin]: <a href="https://github.com/PyO3/maturin">https://github.com/PyO3/maturin</a></p>
<hr /><p><a rel="payment" href="https://afdian.com/a/messense">爱发电上赞助</a></p>]]></content:encoded></item><item><title><![CDATA[数据处理服务在 Kubernetes 上基于 keda 的弹性伸缩实践]]></title><guid>https://messense.me/kubernetes-keda-kafka-autoscaling</guid><link>https://messense.me/kubernetes-keda-kafka-autoscaling</link><dc:creator><![CDATA[messense]]></dc:creator><pubDate>Sun, 20 Dec 2020 09:49:08 +0000</pubDate><content:encoded><![CDATA[<h2>背景</h2>
<p>我们在阿里云托管的 Kubernetes 集群上部署了一套用于工商、司法等公开数据结构化处理的数据链路。该数据处理服务包含了几十个数据类型，每个数据类型作为一个  Deployment 部署，其主要功能是监听 Kafka 队列消息并做 ETL 处理后索引到 Elasticsearch。通常我们会将 Deployment 的副本数设置为其对应的 Kafka 队列的分区数量以保障数据处理的及时性。目前已经部署了 80 多个 Deployment，按照上述部署方式会占用大量的内存资源，但并不是所有的数据类型都会一直有新消息待处理，造成了极大的资源浪费。我们想通过 Kubernetes 的水平自动伸缩器（Horizontal Pod Autoscaler 以下简称 HPA）来根据 Kafka 队列积压的消息数量自动弹性伸缩 Deployment 副本数量，让持续没有新消息的 Deployment 副本数缩容到最小，并在有消息开始积压的时候自动扩容。</p>
<h2>方案调研</h2>
<h3>Custom Metrics</h3>
<p>Kubernetes 原生自带了基于 CPU/Memory 指标的 HPA，显然我们这个场景用不上。也支持<a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-custom-metrics">基于自定义指标的 HPA</a>，对我们这个场景似乎有些用处。</p>
<p>于是找了下阿里云上已有的 custom metrics adapter 发现了 <a href="https://github.com/AliyunContainerService/alibaba-cloud-metrics-adapter">alibaba-cloud-metrics-adapter</a> 这个项目，它支持 Ingress、SLB、CMS、AHAS Sentinel 以及 ARMS Prometheus custom metrics，而我们在用的 Kubernetes 集群已经接入了 ARMS Prometheus 并且通过 <a href="https://github.com/danielqsj/kafka_exporter">kafka-exporter</a> 收集了 Kafka 队列消息积压的指标数据，理论上可以使用。在尝试了 alibaba-cloud-metrics-adapter 之后感觉它真的不是很好用：</p>
<ol>
<li>首先，<a href="https://github.com/AliyunContainerService/alibaba-cloud-metrics-adapter/blob/master/docs/metrics/arms_prometheus.md">文档</a>写得不是很清楚，<code>3.modify configuration</code> 这个步骤都不知道在哪里更新这个配置
<img src="https://i.typlog.com/messense/8391542396_970406.png" alt="image.png" /></li>
<li>对于我们的场景，需要配置很多 Prometheus 的规则对应到每个 Kafka 队列的消息积压数量，很麻烦</li>
</ol>
<p>于是放弃了使用 alibaba-cloud-metrics-adapter 这个方案，同时也否决了类似的 <a href="https://github.com/DirectXMan12/k8s-prometheus-adapter">k8s-prometheus-adpater</a> 项目因为也是要配置非常多的 Prometheus 规则，并且对于我们这些没有经常使用 Prometheus 查询的同学而言 Prometheus 的查询语法有些难掌握，难以维护。</p>
<h3>keda</h3>
<p>偶然在 GitHub Explore 里面发现了 <a href="https://keda.sh/">keda</a> 这个项目，它是由 Microsoft 和 Read Hat 合作开发的一个基于事件触发的 Kubernetes 自动伸缩器，目前是 CNCF sandbox 项目，已经发布到了 2.0 版本。看到它 Scalers 列表中<a href="https://keda.sh/docs/2.0/scalers/apache-kafka/">支持 Apache Kafka</a> 便尝试了一下，在 k8s 集群中<a href="https://keda.sh/docs/2.0/deploy/">安装</a>好 keda 之后，对于想要做 HPA 的 Deployment 只需要<a href="https://keda.sh/docs/2.0/concepts/scaling-deployments/">关联一个 ScaledObject</a> 自定义资源即可。</p>
<p>比如说我们有个 Deployment 叫 <code>brm-index-basic</code>，它运行的应用程序使用的 Kafka 队列 topic 叫 <code>basic</code>，消费者 consumer group 叫 <code>basic</code>，则可以定义如下 <code>ScaledObject</code> 实现 HPA</p>
<p><strong>hpa.yaml</strong></p>
<div class="block-code" data-language="yaml"><div class="highlight"><pre><span></span><code><div class="line"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">keda.sh/v1alpha1</span><span class="w"></span>
</div><div class="line"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ScaledObject</span><span class="w"></span>
</div><div class="line"><span class="nt">metadata</span><span class="p">:</span><span class="w"></span>
</div><div class="line"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">kafka-scaledobject</span><span class="w"></span>
</div><div class="line"><span class="w">  </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">default</span><span class="w"></span>
</div><div class="line"><span class="nt">spec</span><span class="p">:</span><span class="w"></span>
</div><div class="line"><span class="w">  </span><span class="nt">scaleTargetRef</span><span class="p">:</span><span class="w"></span>
</div><div class="line"><span class="w">    </span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">apps/v1</span><span class="w"></span>
</div><div class="line"><span class="w">    </span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Deployment</span><span class="w"></span>
</div><div class="line"><span class="w">    </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">brm-index-basic</span><span class="w"></span>
</div><div class="line"><span class="w">  </span><span class="nt">pollingInterval</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">10</span><span class="w"></span>
</div><div class="line"><span class="w">  </span><span class="nt">minReplicaCount</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1</span><span class="w"></span>
</div><div class="line"><span class="w">  </span><span class="nt">maxReplicaCount</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">20</span><span class="w"></span>
</div><div class="line"><span class="w">  </span><span class="nt">triggers</span><span class="p">:</span><span class="w"></span>
</div><div class="line"><span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">kafka</span><span class="w"></span>
</div><div class="line"><span class="w">      </span><span class="nt">metadata</span><span class="p">:</span><span class="w"></span>
</div><div class="line"><span class="w">        </span><span class="nt">bootstrapServers</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">kafka-server:9092</span><span class="w"></span>
</div><div class="line"><span class="w">        </span><span class="nt">consumerGroup</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">basic</span><span class="w"></span>
</div><div class="line"><span class="w">        </span><span class="nt">topic</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">basic</span><span class="w"></span>
</div><div class="line"><span class="w">        </span><span class="nt">lagThreshold</span><span class="p">:</span><span class="w"> </span><span class="s">&quot;100&quot;</span><span class="w"></span>
</div><div class="line"><span class="w">        </span><span class="nt">offsetResetPolicy</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">latest</span><span class="w"></span>
</div></code></pre></div>
</div>
<ol>
<li>其中，<code>scaleTargetRef</code> 定义了 HPA 要伸缩的对象这里即 <code>brm-index-basic</code> 这个 Deployment</li>
<li><code>pollingInterval</code> 定义了 <code>keda-operator</code> 刷新 triggers 这里即 Kafka 队列消息积压数量的频率，单位为秒，这个数字决定了 HPA 的灵敏度</li>
<li><code>minReplicaCount</code> 和 <code>maxReplicaCount</code> 分别定义了要伸缩的对象的最小和最大副本数量，<code>minReplicaCount</code> 可以为 0 即缩容到没有副本，比方说 Kafka 队列一直没有新消息就可以完全缩容，到有新消息进来的时候 keda 又会自动扩容，上述的 <code>pollingInterval</code> 就会影响这里从 0 到扩容的时间间隔，<code>maxReplicaCount</code> 我们会设置为 Kafka 队列 topic 的分区数量</li>
<li><code>triggers</code> 中则配置了 Kafka 集群的连接信息和关联的 topic/consumer group 名字以及消息积压阈值 <code>lagThreshold</code>，<code>lagThreshold</code> 越小 HPA 伸缩时越激进，越大越保守</li>
<li><code>offsetResetPolicy</code> 主要是为 Kafka 消息队列还没有过消费者的时候设计的，一般设置成 <code>latest</code> 即可，具体可以参考官方文档：<a href="https://keda.sh/docs/2.0/scalers/apache-kafka/#new-consumers-and-offset-reset-policy">https://keda.sh/docs/2.0/scalers/apache-kafka/#new-consumers-and-offset-reset-policy</a></li>
</ol>
<p><code>kubectl apply -f hpa.yaml</code> 后即为 <code>brm-index-basic</code> Deployment 增加了基于 keda 的 HPA，通过 <code>kubectl get hpa</code> 可以查看 HPA 的信息：</p>
<div class="block-code" data-language="bash"><div class="highlight"><pre><span></span><code><div class="line">$ kubectl get hpa
</div><div class="line">NAME                                                    REFERENCE                                        TARGETS               MINPODS   MAXPODS   REPLICAS   AGE
</div><div class="line">keda-hpa-kafka-scaledobject-basic                       Deployment/brm-index-basic                       39500m/100 <span class="o">(</span>avg<span class="o">)</span>      <span class="m">1</span>         <span class="m">20</span>        <span class="m">2</span>          5d22h
</div></code></pre></div>
</div>
<p>这里需要注意的是 HPA TARGETS 里面的带 <code>m</code> 结尾的数字单位不是百万而是千分之一，即这里平均积压数量是 39.5，参考文档：<a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#appendix-quantities">https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#appendix-quantities</a></p>
<h2>总结</h2>
<p>最终我们给大部分数据索引部署都增加了基于 keda 的自动水平伸缩，通过监控 Kafka 队列未消费消息数量自动伸缩索引任务容器数量，没有索引消息的时候自动缩容到最少 0 个容器，大大降低日常 Kubernetes 集群 CPU/内存资源占用，同时也建立了在大量索引任务进来的时候快速扩容的能力。</p>
<hr /><p><a rel="payment" href="https://afdian.com/a/messense">爱发电上赞助</a></p>]]></content:encoded></item><item><title><![CDATA[在 Cython 项目中使用 abseil-cpp]]></title><guid>https://messense.me/use-abseil-cpp-with-cython</guid><link>https://messense.me/use-abseil-cpp-with-cython</link><dc:creator><![CDATA[messense]]></dc:creator><pubDate>Sun, 10 May 2020 08:54:24 +0000</pubDate><content:encoded><![CDATA[<p>上一篇文章介绍了<a href="https://messense.me/build-python-c-extensions-with-cmake">使用 CMake 构建 Python C/C++ 扩展</a>的两个方案，也提到了之前在工作中做这个的主要目的是为了能够使用 <a href="https://abseil.io/docs/cpp">abseil-cpp</a> 的 Swiss Tables 优化性能，那么这篇文章就来简单介绍一下如何在 Cython 项目中使用 abseil-cpp.</p>
<h2>项目目录结构</h2>
<div class="block-code" data-language="bash"><div class="highlight"><pre><span></span><code><div class="line">$ tree -L <span class="m">2</span> .
</div><div class="line">.
</div><div class="line">├── CMakeLists.txt        <span class="c1"># 项目主 CMake 配置</span>
</div><div class="line">├── absl.pxd              <span class="c1"># abseil-cpp 接口的部分 Cython 定义文件</span>
</div><div class="line">├── hello                 <span class="c1"># hello Python 库</span>
</div><div class="line">│   ├── CMakeLists.txt    <span class="c1"># Python C++ 扩展 CMake 配置</span>
</div><div class="line">│   ├── __init__.py
</div><div class="line">│   └── _hello.pyx        <span class="c1"># Cython 文件</span>
</div><div class="line">├── pyproject.toml
</div><div class="line">├── setup.py
</div><div class="line">├── src
</div><div class="line">│   ├── CMakeLists.txt    <span class="c1"># 项目 C/C++ 代码 CMake 配置</span>
</div><div class="line">│   ├── absl.cc
</div><div class="line">│   └── absl.h
</div><div class="line">├── test_hello_cython.py  <span class="c1"># 测试代码</span>
</div><div class="line">├── vendor                <span class="c1"># C/C++ 第三方库放置路径</span>
</div><div class="line">│   └── abseil-cpp        <span class="c1"># abseil-cpp Git submodule</span>
</div><div class="line">└── venv                  <span class="c1"># Python virtualenv</span>
</div></code></pre></div>
</div>
<p>该示例项目的代码托管在 GitHub： <a href="https://github.com/messense/cython-abseil-sample">https://github.com/messense/cython-abseil-sample</a></p>
<h2>接入 abseil-cpp</h2>
<p>可以直接把 abseil-cpp 作为一个 Git submodule 导入到自己的项目中</p>
<div class="block-code" data-language="bash"><div class="highlight"><pre><span></span><code><div class="line">git submodule add --depth <span class="m">50</span> https://github.com/abseil/abseil-cpp.git vendor/abseil-cpp
</div></code></pre></div>
</div>
<p>这里我把它放到了 <code>vendor/abseil-cpp</code> 目录中，然后可以在项目主 <code>CMakeLists.txt</code> 文件中配置依赖，<a href="https://abseil.io/docs/cpp/quickstart-cmake#creating-your-cmakeliststxt-file">官方文档</a> 上有更详细的说明。</p>
<div class="block-code" data-language="cmake"><div class="highlight"><pre><span></span><code><div class="line"><span class="nb">cmake_minimum_required</span><span class="p">(</span><span class="s">VERSION</span><span class="w"> </span><span class="s">3.5.0</span><span class="p">)</span>
</div><div class="line"><span class="nb">project</span><span class="p">(</span><span class="s">hello</span><span class="p">)</span>
</div><div class="line">
</div><div class="line"><span class="nb">set</span> <span class="p">(</span><span class="s">CMAKE_POSITION_INDEPENDENT_CODE</span><span class="w"> </span><span class="s">ON</span><span class="p">)</span>
</div><div class="line">
</div><div class="line"><span class="c"># Abseil requires C++11</span>
</div><div class="line"><span class="nb">set</span><span class="p">(</span><span class="s">CMAKE_CXX_STANDARD</span><span class="w"> </span><span class="s">11</span><span class="p">)</span>
</div><div class="line">
</div><div class="line"><span class="c"># Process Abseil&#39;s CMake build system</span>
</div><div class="line"><span class="nb">add_subdirectory</span><span class="p">(</span><span class="s">vendor/abseil-cpp</span><span class="p">)</span>
</div><div class="line"><span class="nb">include_directories</span><span class="p">(</span>
</div><div class="line"><span class="w">  </span><span class="s">/usr/local/include</span>
</div><div class="line"><span class="w">  </span><span class="o">${</span><span class="nv">CMAKE_CURRENT_SOURCE_DIR</span><span class="o">}</span><span class="s">/vendor/abseil-cpp</span>
</div><div class="line"><span class="w">  </span><span class="o">${</span><span class="nv">CMAKE_CURRENT_SOURCE_DIR</span><span class="o">}</span><span class="s">/src</span><span class="p">)</span>
</div><div class="line">
</div><div class="line"><span class="nb">add_subdirectory</span><span class="p">(</span><span class="s">src</span><span class="p">)</span>
</div><div class="line"><span class="nb">add_subdirectory</span><span class="p">(</span><span class="s">hello</span><span class="p">)</span>
</div></code></pre></div>
</div>
<p>上面的配置把 <code>src/</code> 和 <code>hello/</code> 目录也作为 CMake 子目录加入了配置，下面分别介绍一下。</p>
<h2>src 目录 - C/C++ 模块</h2>
<p>src 目录主要存放了该项目点 C/C++ 代码，这部分代码和是 Python 无关的。</p>
<p><strong>CMakeLists.txt</strong></p>
<div class="block-code" data-language="cmake"><div class="highlight"><pre><span></span><code><div class="line"><span class="nb">set</span><span class="p">(</span><span class="s">SOURCES</span><span class="w"> </span><span class="s">absl.cc</span><span class="p">)</span>
</div><div class="line"><span class="nb">add_library</span><span class="p">(</span><span class="s">hello_absl</span><span class="w"> </span><span class="s">STATIC</span><span class="w"> </span><span class="o">${</span><span class="nv">SOURCES</span><span class="o">}</span><span class="p">)</span>
</div><div class="line"><span class="nb">target_link_libraries</span><span class="p">(</span><span class="s">hello_absl</span><span class="w"> </span><span class="s">absl::flat_hash_map</span><span class="p">)</span>
</div></code></pre></div>
</div>
<p>这里的构建配置非常简单，只是构建了一个静态的 <code>hello_absl</code> target 并链接 <code>absl::flat_hash_map</code> 库，这个 target 暴露的 public API 只有一个返回值类型为 <code>absl::flat_hash_map</code> 的函数 <code>fn_return_flat_hash_map</code>，定义在 <code>src/absl.h</code> 文件中：</p>
<div class="block-code" data-language="cpp"><div class="highlight"><pre><span></span><code><div class="line"><span class="cp">#include</span><span class="w"> </span><span class="cpf">&lt;string&gt;</span><span class="cp"></span>
</div><div class="line"><span class="cp">#include</span><span class="w"> </span><span class="cpf">&quot;absl/container/flat_hash_map.h&quot;</span><span class="cp"></span>
</div><div class="line">
</div><div class="line"><span class="n">absl</span><span class="o">::</span><span class="n">flat_hash_map</span><span class="o">&lt;</span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">&gt;</span><span class="w"> </span><span class="n">fn_return_flat_hash_map</span><span class="p">(</span><span class="kt">void</span><span class="p">);</span><span class="w"></span>
</div></code></pre></div>
</div>
<p>其实现也非常简单 <code>absl.cc</code>:</p>
<div class="block-code" data-language="cpp"><div class="highlight"><pre><span></span><code><div class="line"><span class="cp">#include</span><span class="w"> </span><span class="cpf">&quot;absl.h&quot;</span><span class="cp"></span>
</div><div class="line">
</div><div class="line"><span class="n">absl</span><span class="o">::</span><span class="n">flat_hash_map</span><span class="o">&lt;</span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">&gt;</span><span class="w"> </span><span class="n">fn_return_flat_hash_map</span><span class="p">(</span><span class="kt">void</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
</div><div class="line"><span class="w">    </span><span class="n">absl</span><span class="o">::</span><span class="n">flat_hash_map</span><span class="o">&lt;</span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">&gt;</span><span class="w"> </span><span class="n">numbers</span><span class="w"> </span><span class="o">=</span><span class="w"></span>
</div><div class="line"><span class="w">        </span><span class="p">{{</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;one&quot;</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;two&quot;</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;three&quot;</span><span class="p">}};</span><span class="w"></span>
</div><div class="line"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="n">numbers</span><span class="p">;</span><span class="w"></span>
</div><div class="line"><span class="p">}</span><span class="w"></span>
</div></code></pre></div>
</div>
<h2>hello 目录 - Cython 模块</h2>
<p>hello 目录存放的是 Python 模块代码，包含一个 <code>_hello.pyx</code> 的 Cython 模块。</p>
<p><strong>CMakeLists.txt</strong></p>
<div class="block-code" data-language="cmake"><div class="highlight"><pre><span></span><code><div class="line"><span class="nb">if</span><span class="p">(</span><span class="s">SKBUILD</span><span class="p">)</span>
</div><div class="line"><span class="w">  </span><span class="nb">message</span><span class="p">(</span><span class="s">STATUS</span><span class="w"> </span><span class="s2">&quot;The project is built using scikit-build&quot;</span><span class="p">)</span>
</div><div class="line"><span class="w">  </span><span class="nb">find_package</span><span class="p">(</span><span class="s">PythonExtensions</span><span class="w"> </span><span class="s">REQUIRED</span><span class="p">)</span>
</div><div class="line"><span class="w">  </span><span class="nb">find_package</span><span class="p">(</span><span class="s">Cython</span><span class="w"> </span><span class="s">REQUIRED</span><span class="p">)</span>
</div><div class="line">
</div><div class="line"><span class="w">  </span><span class="nb">include_directories</span><span class="p">(</span>
</div><div class="line"><span class="w">    </span><span class="s">/usr/local/include</span>
</div><div class="line"><span class="w">    </span><span class="o">${</span><span class="nv">PYTHON_INCLUDE_DIRS</span><span class="o">}</span>
</div><div class="line"><span class="w">    </span><span class="o">${</span><span class="nv">CMAKE_CURRENT_SOURCE_DIR</span><span class="o">}</span><span class="s">/vendor/abseil-cpp</span><span class="p">)</span>
</div><div class="line">
</div><div class="line"><span class="w">  </span><span class="nb">add_cython_target</span><span class="p">(</span><span class="s">_hello</span><span class="w"> </span><span class="s">CXX</span><span class="p">)</span>
</div><div class="line"><span class="w">  </span><span class="nb">add_library</span><span class="p">(</span><span class="s">_hello</span><span class="w"> </span><span class="s">MODULE</span><span class="w"> </span><span class="o">${</span><span class="nv">_hello</span><span class="o">}</span><span class="p">)</span>
</div><div class="line"><span class="w">  </span><span class="nb">target_link_libraries</span><span class="p">(</span><span class="s">_hello</span><span class="w"> </span><span class="s">hello_absl</span><span class="p">)</span>
</div><div class="line"><span class="w">  </span><span class="nb">python_extension_module</span><span class="p">(</span><span class="s">_hello</span><span class="p">)</span>
</div><div class="line">
</div><div class="line"><span class="w">  </span><span class="nb">install</span><span class="p">(</span><span class="s">TARGETS</span><span class="w"> </span><span class="s">_hello</span><span class="w"> </span><span class="s">LIBRARY</span><span class="w"> </span><span class="s">DESTINATION</span><span class="w"> </span><span class="s">hello</span><span class="p">)</span>
</div><div class="line"><span class="nb">endif</span><span class="p">()</span>
</div></code></pre></div>
</div>
<ol>
<li>增加 <code>SKBUILD</code> 变量的判断，只有在使用 scikit-build 构建的时候才会去构建 Python 扩展，这样如果你的项目中的 C/C++ 部分不依赖 Python 也可以运行、测试或者 benchmark 的话，会比较方便</li>
<li>增加了 <code>find_package(Cython REQUIRED)</code> 和 <code>add_cython_target(_hello CXX)</code> 配置 Cython 模块支持</li>
</ol>
<p><strong>_hello.pyx</strong></p>
<div class="block-code" data-language="python"><div class="highlight"><pre><span></span><code><div class="line"><span class="kn">from</span> <span class="nn">libcpp.string</span> <span class="n">cimport</span> <span class="n">string</span>
</div><div class="line">
</div><div class="line"><span class="kn">from</span> <span class="nn">absl</span> <span class="n">cimport</span> <span class="n">flat_hash_map</span>
</div><div class="line">
</div><div class="line"><span class="n">cdef</span> <span class="n">extern</span> <span class="kn">from</span> <span class="s2">&quot;absl.h&quot;</span><span class="p">:</span>
</div><div class="line">    <span class="n">cdef</span> <span class="n">flat_hash_map</span><span class="p">[</span><span class="nb">int</span><span class="p">,</span> <span class="n">string</span><span class="p">]</span> <span class="n">fn_return_flat_hash_map</span><span class="p">()</span> <span class="n">nogil</span>
</div><div class="line">
</div><div class="line"><span class="n">cpdef</span> <span class="nb">dict</span> <span class="n">hello_absl</span><span class="p">():</span>
</div><div class="line">    <span class="n">cdef</span> <span class="n">flat_hash_map</span><span class="p">[</span><span class="nb">int</span><span class="p">,</span> <span class="n">string</span><span class="p">]</span> <span class="n">a_map</span>
</div><div class="line">    <span class="k">with</span> <span class="n">nogil</span><span class="p">:</span>
</div><div class="line">      <span class="n">a_map</span> <span class="o">=</span> <span class="n">fn_return_flat_hash_map</span><span class="p">()</span>
</div><div class="line">    <span class="k">return</span> <span class="p">{</span><span class="n">it</span><span class="o">.</span><span class="n">first</span><span class="p">:</span> <span class="n">it</span><span class="o">.</span><span class="n">second</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span> <span class="k">for</span> <span class="n">it</span> <span class="ow">in</span> <span class="n">a_map</span><span class="p">}</span>
</div></code></pre></div>
</div>
<ol>
<li>从 Cython 自带的 <code>libcpp</code> 标准库导入了 <code>std::string</code></li>
<li>从我们定义的 <code>absl.pxd</code> 导入了 <code>flat_hash_map</code>，<code>absl.pxd</code> 详情下面再解释</li>
<li>使用 <code>cdef extern</code> 定义了上述 <code>src</code> 目录中 C++ 模块定义的 <code>fn_return_flat_hash_map</code> 函数原型</li>
<li>在 <code>cpdef dict hello_absl</code> 函数中调用 <code>fn_return_flat_hash_map</code> C++ 函数并转换成 Python dict 返回</li>
</ol>
<p>这里有两个问题需要解答：</p>
<ol>
<li><code>from absl cimport flat_hash_map</code> 如何工作的?</li>
<li><code>flat_hash_map</code> 转换成 Python dict 为什么不能像 <code>libcpp</code> 的 <code>unordered_map</code> 一样直接赋值给 dict 类型的变量自动转换而需要迭代手动生成一个 dict?</li>
</ol>
<p>问题 1 我们需要了解一下 Cython 是如何调用 C/C++ 函数的，官方文档中关于 <a href="https://cython.readthedocs.io/en/latest/src/tutorial/pxd_files.html">pxd 文件的文档</a>有如下解释</p>
<div class="blockquote"><blockquote><p>Cython uses <code>.pxd</code> files which work like C header files – they contain Cython declarations (and sometimes code sections) which are only meant for inclusion by Cython modules. A pxd file is imported into a pyx module by using the cimport keyword.</p>
</blockquote></div>
<p>为什么 <code>from libcpp.string import string</code> 不需要我们提供 pxd 文件呢？这是因为 <a href="https://cython.readthedocs.io/en/latest/src/userguide/wrapping_CPlusPlus.html#standard-library">Cython 已经默认包含了 libcpp 的接口定义</a>，但是 abseil-cpp 的接口定义 Cython 默认是没有的，所以需要提供我们，因为我们这里只用到了 abseil-cpp 的 <code>flat_hash_map</code> 类，故 <code>absl.pxd</code> 中目前只有它的定义，如果以后用到其他的类，也可以在里面增加定义。</p>
<p><strong>absl.pxd</strong></p>
<p><code>absl.pxd</code> 的内容是从 Cython 自带的 <code>libcpp</code> 代码中的 <code>unordered_map.pxd</code> 代码修改的，diff 一下这两个文件</p>
<div class="block-code" data-language="diff"><div class="highlight"><pre><span></span><code><div class="line"><span class="gd">--- unordered_map.pxd2020-05-10 16:29:28.000000000 +0800</span><span class="w"></span>
</div><div class="line"><span class="gi">+++ absl.pxd2020-05-10 14:54:27.000000000 +0800</span><span class="w"></span>
</div><div class="line"><span class="gu">@@ -1,7 +1,7 @@</span><span class="w"></span>
</div><div class="line"><span class="gd">-from .utility cimport pair</span><span class="w"></span>
</div><div class="line"><span class="gi">+from libcpp.utility cimport pair</span><span class="w"></span>
</div><div class="line"><span class="w"> </span>
</div><div class="line"><span class="gd">-cdef extern from &quot;&lt;unordered_map&gt;&quot; namespace &quot;std&quot; nogil:</span><span class="w"></span>
</div><div class="line"><span class="gd">-    cdef cppclass unordered_map[T, U]:</span><span class="w"></span>
</div><div class="line"><span class="gi">+cdef extern from &quot;absl/container/flat_hash_map.h&quot; namespace &quot;absl&quot; nogil:</span><span class="w"></span>
</div><div class="line"><span class="gi">+    cdef cppclass flat_hash_map[T, U]:</span><span class="w"></span>
</div><div class="line"><span class="w"> </span>        ctypedef T key_type<span class="w"></span>
</div><div class="line"><span class="w"> </span>        ctypedef U mapped_type<span class="w"></span>
</div><div class="line"><span class="w"> </span>        ctypedef pair[const T, U] value_type<span class="w"></span>
</div><div class="line"><span class="gu">@@ -21,17 +21,17 @@</span><span class="w"></span>
</div><div class="line"><span class="w"> </span>            pass<span class="w"></span>
</div><div class="line"><span class="w"> </span>        cppclass const_reverse_iterator(reverse_iterator):<span class="w"></span>
</div><div class="line"><span class="w"> </span>            pass<span class="w"></span>
</div><div class="line"><span class="gd">-        unordered_map() except +</span><span class="w"></span>
</div><div class="line"><span class="gd">-        unordered_map(unordered_map&amp;) except +</span><span class="w"></span>
</div><div class="line"><span class="gd">-        #unordered_map(key_compare&amp;)</span><span class="w"></span>
</div><div class="line"><span class="gi">+        flat_hash_map() except +</span><span class="w"></span>
</div><div class="line"><span class="gi">+        flat_hash_map(flat_hash_map&amp;) except +</span><span class="w"></span>
</div><div class="line"><span class="gi">+        #flat_hash_map(key_compare&amp;)</span><span class="w"></span>
</div><div class="line"><span class="w"> </span>        U&amp; operator[](T&amp;)<span class="w"></span>
</div><div class="line"><span class="gd">-        #unordered_map&amp; operator=(unordered_map&amp;)</span><span class="w"></span>
</div><div class="line"><span class="gd">-        bint operator==(unordered_map&amp;, unordered_map&amp;)</span><span class="w"></span>
</div><div class="line"><span class="gd">-        bint operator!=(unordered_map&amp;, unordered_map&amp;)</span><span class="w"></span>
</div><div class="line"><span class="gd">-        bint operator&lt;(unordered_map&amp;, unordered_map&amp;)</span><span class="w"></span>
</div><div class="line"><span class="gd">-        bint operator&gt;(unordered_map&amp;, unordered_map&amp;)</span><span class="w"></span>
</div><div class="line"><span class="gd">-        bint operator&lt;=(unordered_map&amp;, unordered_map&amp;)</span><span class="w"></span>
</div><div class="line"><span class="gd">-        bint operator&gt;=(unordered_map&amp;, unordered_map&amp;)</span><span class="w"></span>
</div><div class="line"><span class="gi">+        #flat_hash_map&amp; operator=(flat_hash_map&amp;)</span><span class="w"></span>
</div><div class="line"><span class="gi">+        bint operator==(flat_hash_map&amp;, flat_hash_map&amp;)</span><span class="w"></span>
</div><div class="line"><span class="gi">+        bint operator!=(flat_hash_map&amp;, flat_hash_map&amp;)</span><span class="w"></span>
</div><div class="line"><span class="gi">+        bint operator&lt;(flat_hash_map&amp;, flat_hash_map&amp;)</span><span class="w"></span>
</div><div class="line"><span class="gi">+        bint operator&gt;(flat_hash_map&amp;, flat_hash_map&amp;)</span><span class="w"></span>
</div><div class="line"><span class="gi">+        bint operator&lt;=(flat_hash_map&amp;, flat_hash_map&amp;)</span><span class="w"></span>
</div><div class="line"><span class="gi">+        bint operator&gt;=(flat_hash_map&amp;, flat_hash_map&amp;)</span><span class="w"></span>
</div><div class="line"><span class="w"> </span>        U&amp; at(const T&amp;)<span class="w"></span>
</div><div class="line"><span class="w"> </span>        const U&amp; const_at &quot;at&quot;(const T&amp;)<span class="w"></span>
</div><div class="line"><span class="w"> </span>        iterator begin()<span class="w"></span>
</div><div class="line"><span class="gu">@@ -43,8 +43,8 @@</span><span class="w"></span>
</div><div class="line"><span class="w"> </span>        const_iterator const_end &quot;end&quot;()<span class="w"></span>
</div><div class="line"><span class="w"> </span>        pair[iterator, iterator] equal_range(T&amp;)<span class="w"></span>
</div><div class="line"><span class="w"> </span>        pair[const_iterator, const_iterator] const_equal_range &quot;equal_range&quot;(const T&amp;)<span class="w"></span>
</div><div class="line"><span class="gd">-        iterator erase(iterator)</span><span class="w"></span>
</div><div class="line"><span class="gd">-        iterator erase(iterator, iterator)</span><span class="w"></span>
</div><div class="line"><span class="gi">+        void erase(iterator)</span><span class="w"></span>
</div><div class="line"><span class="gi">+        void erase(iterator, iterator)</span><span class="w"></span>
</div><div class="line"><span class="w"> </span>        size_t erase(T&amp;)<span class="w"></span>
</div><div class="line"><span class="w"> </span>        iterator find(T&amp;)<span class="w"></span>
</div><div class="line"><span class="w"> </span>        const_iterator const_find &quot;find&quot;(T&amp;)<span class="w"></span>
</div><div class="line"><span class="gu">@@ -60,7 +60,7 @@</span><span class="w"></span>
</div><div class="line"><span class="w"> </span>        reverse_iterator rend()<span class="w"></span>
</div><div class="line"><span class="w"> </span>        const_reverse_iterator const_rend &quot;rend&quot;()<span class="w"></span>
</div><div class="line"><span class="w"> </span>        size_t size()<span class="w"></span>
</div><div class="line"><span class="gd">-        void swap(unordered_map&amp;)</span><span class="w"></span>
</div><div class="line"><span class="gi">+        void swap(flat_hash_map&amp;)</span><span class="w"></span>
</div><div class="line"><span class="w"> </span>        iterator upper_bound(T&amp;)<span class="w"></span>
</div><div class="line"><span class="w"> </span>        const_iterator const_upper_bound &quot;upper_bound&quot;(T&amp;)<span class="w"></span>
</div><div class="line"><span class="w"> </span>        #value_compare value_comp()<span class="w"></span>
</div></code></pre></div>
</div>
<p>可以发现，除了将 <code>unordered_map</code> 名字替换成 <code>flat_hash_map</code> 和修改 <code>extern from</code> 路径外，只有几个 <code>erase</code> 方法的返回值类型不一样，<code>unordered_map.erase</code> 会返回一个新的 <code>iterator</code> 而 <code>flat_hash_map.erase</code> 返回值类型为 <code>void</code>，这是因为 abseil-cpp 的 <code>flat_hash_map</code> 为了优化 erase 方法的性能和 C++ STL 的 <code>unordered_map</code> erase 方法行为有所不同导致的：</p>
<div class="blockquote"><blockquote><p>Guarantees an `O(1)`` erase method by returning void instead of an iterator.</p>
</blockquote></div>
<p>问题 2 则是因为 Cython 编译器对 libcpp 有特殊支持会生成很多转换函数，而我们自行提供的 <code>.pxd</code> 文件则没有这种特殊待遇导致的。有兴趣的可以看一下 <code>Cython.Compiler.PyrexTypes</code> 模块中的 <a href="https://github.com/cython/cython/blob/5b6bbdab8387666bce5e11975e1e0baa0081a534/Cython/Compiler/PyrexTypes.py#L470">CTypedefType.create_to_py_utility_code</a> 方法代码。</p>
<p>本文只给出了一个 C++ 代码返回值类型为 abseil-cpp 类型的例子，实际上也可以扩展为函数参数类型为 abseil-cpp 或者其他任何 C/C++ 第三方库的类型，这里不再赘述。</p>
<h2>测试</h2>
<p><code>test_hello_cython.py</code> 文件中编写了一个 <code>test_hello_absl</code> 测试用例：</p>
<div class="block-code" data-language="python"><div class="highlight"><pre><span></span><code><div class="line"><span class="kn">import</span> <span class="nn">hello</span>
</div><div class="line">
</div><div class="line">
</div><div class="line"><span class="k">def</span> <span class="nf">test_hello_absl</span><span class="p">():</span>
</div><div class="line">    <span class="n">ret</span> <span class="o">=</span> <span class="n">hello</span><span class="o">.</span><span class="n">hello_absl</span><span class="p">()</span>
</div><div class="line">    <span class="k">assert</span> <span class="n">ret</span> <span class="o">==</span> <span class="p">{</span><span class="mi">1</span><span class="p">:</span> <span class="s2">&quot;one&quot;</span><span class="p">,</span> <span class="mi">2</span><span class="p">:</span> <span class="s2">&quot;two&quot;</span><span class="p">,</span> <span class="mi">3</span><span class="p">:</span> <span class="s2">&quot;three&quot;</span><span class="p">}</span>
</div></code></pre></div>
</div>
<p>运行 <code>pytest -v</code> 可以确认测试正常：</p>
<div class="block-code" data-language="bash"><div class="highlight"><pre><span></span><code><div class="line">$ pytest -v
</div><div class="line">test_hello_cython.py::test_hello_absl PASSED
</div></code></pre></div>
</div>
<hr /><p><a rel="payment" href="https://afdian.com/a/messense">爱发电上赞助</a></p>]]></content:encoded></item><item><title><![CDATA[使用 CMake 构建 Python C/C++ 扩展]]></title><guid>https://messense.me/build-python-c-extensions-with-cmake</guid><link>https://messense.me/build-python-c-extensions-with-cmake</link><dc:creator><![CDATA[messense]]></dc:creator><pubDate>Sun, 03 May 2020 08:50:52 +0000</pubDate><content:encoded><![CDATA[<p>众所周知，Python 语言的性能相比其他语言如 C/C++ 等要弱很多，所以当我们需要高性能的时候往往借助于 Python 的 C/C++ 扩展或者 <a href="https://cython.org/" title="Cython">Cython</a>。</p>
<p>通常构建 Python C/C++ 扩展会使用 <a href="https://docs.python.org/3.8/distutils/index.html#distutils-index" title="distutils">distutils</a> 的 <a href="https://docs.python.org/3.8/distutils/apiref.html#distutils.core.Extension" title="`Extension`"><code>Extension</code></a> 类，需要在 <code>setup.py</code> 中配置头文件包含路径 <code>include_dirs</code>、C/C++ 源文件路径 <code>sources</code> 等，比如下面这个<a href="https://docs.python.org/3.8/extending/building.html#building-c-and-c-extensions-with-distutils" title="Python 官方文档上的例子">Python 官方文档上的例子</a>：</p>
<div class="block-code" data-language="python"><div class="highlight"><pre><span></span><code><div class="line"><span class="kn">from</span> <span class="nn">distutils.core</span> <span class="kn">import</span> <span class="n">setup</span><span class="p">,</span> <span class="n">Extension</span>
</div><div class="line">
</div><div class="line"><span class="n">module1</span> <span class="o">=</span> <span class="n">Extension</span><span class="p">(</span>
</div><div class="line">    <span class="s1">&#39;demo&#39;</span><span class="p">,</span>
</div><div class="line">    <span class="n">define_macros</span><span class="o">=</span><span class="p">[</span>
</div><div class="line">        <span class="p">(</span><span class="s1">&#39;MAJOR_VERSION&#39;</span><span class="p">,</span> <span class="s1">&#39;1&#39;</span><span class="p">),</span>
</div><div class="line">        <span class="p">(</span><span class="s1">&#39;MINOR_VERSION&#39;</span><span class="p">,</span> <span class="s1">&#39;0&#39;</span><span class="p">)</span>
</div><div class="line">    <span class="p">],</span>
</div><div class="line">    <span class="n">include_dirs</span><span class="o">=</span><span class="p">[</span><span class="s1">&#39;/usr/local/include&#39;</span><span class="p">],</span>
</div><div class="line">    <span class="n">libraries</span> <span class="o">=</span><span class="p">[</span><span class="s1">&#39;tcl83&#39;</span><span class="p">],</span>
</div><div class="line">    <span class="n">library_dirs</span><span class="o">=</span><span class="p">[</span><span class="s1">&#39;/usr/local/lib&#39;</span><span class="p">],</span>
</div><div class="line">    <span class="n">sources</span><span class="o">=</span><span class="p">[</span><span class="s1">&#39;demo.c&#39;</span><span class="p">]</span>
</div><div class="line"><span class="p">)</span>
</div><div class="line">
</div><div class="line"><span class="n">setup</span><span class="p">(</span>
</div><div class="line">    <span class="n">name</span><span class="o">=</span><span class="s1">&#39;PackageName&#39;</span><span class="p">,</span>
</div><div class="line">    <span class="n">version</span><span class="o">=</span><span class="s1">&#39;1.0&#39;</span><span class="p">,</span>
</div><div class="line">    <span class="n">description</span><span class="o">=</span><span class="s1">&#39;This is a demo package&#39;</span><span class="p">,</span>
</div><div class="line">    <span class="n">author</span><span class="o">=</span><span class="s1">&#39;Martin v. Loewis&#39;</span><span class="p">,</span>
</div><div class="line">    <span class="n">author_email</span><span class="o">=</span><span class="s1">&#39;martin@v.loewis.de&#39;</span><span class="p">,</span>
</div><div class="line">    <span class="n">url</span><span class="o">=</span><span class="s1">&#39;https://docs.python.org/extending/building&#39;</span><span class="p">,</span>
</div><div class="line">    <span class="n">long_description</span><span class="o">=</span><span class="s1">&#39;&#39;&#39;</span>
</div><div class="line"><span class="s1">This is really just a demo package.</span>
</div><div class="line"><span class="s1">&#39;&#39;&#39;</span><span class="p">,</span>
</div><div class="line">    <span class="n">ext_modules</span><span class="o">=</span><span class="p">[</span><span class="n">module1</span><span class="p">]</span>
</div><div class="line"><span class="p">)</span>
</div></code></pre></div>
</div>
<p>这种方式对于绝大多数简单的项目应该是足够了，而当你需要用到一些 C/C++ 第三方库的时候可能会遇到因为某些原因需要将三方库的源码和项目源码一起进行编译的情况（比如 <a href="https://abseil.io/docs/cpp" title="abseil-cpp">abseil-cpp</a>），这个情况下往往会遇到 C/C++ 依赖管理的问题，<a href="https://cmake.org/" title="CMake">CMake</a> 则是常用的 C/C++ 依赖管理工具，本文将总结、分享一下使用 CMake 来构建 Python C/C++ 扩展的方案。</p>
<h2>调研可选方案</h2>
<p>首先来看一下 CMake 项目本身一般是如何构建的，一般 CMake 项目都会在项目根目录下有个 <code>CMakeLists.txt</code> 的 CMake 项目定义文件，构建方式通常如下：</p>
<div class="block-code" data-language="bash"><div class="highlight"><pre><span></span><code><div class="line">mkdir build
</div><div class="line"><span class="nb">cd</span> build
</div><div class="line">cmake ..
</div><div class="line">make
</div></code></pre></div>
</div>
<p>那基本的思路就是在 Python 包构建过程中（<code>pip install</code> 或者 <code>python setup.py install</code> 等）调用上述命令完成扩展构建。通过 Google 搜索可以发现，一个方案是通过继承 distutils 的 <code>Extension</code> 来手工实现，另一个方案则是用别人写好的现成的封装库 <a href="https://github.com/scikit-build/scikit-build" title="scikit-build">scikit-build</a>。</p>
<h2>方案一 distutils CMake extension</h2>
<p>这个方案有个现成的例子，<a href="https://github.com/pybind/cmake_example" title="pybind11 的 CMake 示例项目">pybind11 的 CMake 示例项目</a>，BTW，<a href="https://github.com/pybind/pybind11" title="pybind11">pybind11</a> 也是一个写 Python C++ 扩展的项目。看一下它的 <a href="https://github.com/pybind/cmake_example/blob/master/setup.py" title="pybind11 cmake_example setup.py"><code>setup.py</code></a> 的代码：</p>
<div class="block-code" data-language="python"><div class="highlight"><pre><span></span><code><div class="line"><span class="kn">import</span> <span class="nn">os</span>
</div><div class="line"><span class="kn">import</span> <span class="nn">re</span>
</div><div class="line"><span class="kn">import</span> <span class="nn">sys</span>
</div><div class="line"><span class="kn">import</span> <span class="nn">platform</span>
</div><div class="line"><span class="kn">import</span> <span class="nn">subprocess</span>
</div><div class="line">
</div><div class="line"><span class="kn">from</span> <span class="nn">setuptools</span> <span class="kn">import</span> <span class="n">setup</span><span class="p">,</span> <span class="n">Extension</span>
</div><div class="line"><span class="kn">from</span> <span class="nn">setuptools.command.build_ext</span> <span class="kn">import</span> <span class="n">build_ext</span>
</div><div class="line"><span class="kn">from</span> <span class="nn">distutils.version</span> <span class="kn">import</span> <span class="n">LooseVersion</span>
</div><div class="line">
</div><div class="line">
</div><div class="line"><span class="k">class</span> <span class="nc">CMakeExtension</span><span class="p">(</span><span class="n">Extension</span><span class="p">):</span>
</div><div class="line">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">sourcedir</span><span class="o">=</span><span class="s1">&#39;&#39;</span><span class="p">):</span>
</div><div class="line">        <span class="n">Extension</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">sources</span><span class="o">=</span><span class="p">[])</span>
</div><div class="line">        <span class="bp">self</span><span class="o">.</span><span class="n">sourcedir</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">abspath</span><span class="p">(</span><span class="n">sourcedir</span><span class="p">)</span>
</div><div class="line">
</div><div class="line">
</div><div class="line"><span class="k">class</span> <span class="nc">CMakeBuild</span><span class="p">(</span><span class="n">build_ext</span><span class="p">):</span>
</div><div class="line">    <span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
</div><div class="line">        <span class="k">try</span><span class="p">:</span>
</div><div class="line">            <span class="n">out</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">check_output</span><span class="p">([</span><span class="s1">&#39;cmake&#39;</span><span class="p">,</span> <span class="s1">&#39;--version&#39;</span><span class="p">])</span>
</div><div class="line">        <span class="k">except</span> <span class="ne">OSError</span><span class="p">:</span>
</div><div class="line">            <span class="k">raise</span> <span class="ne">RuntimeError</span><span class="p">(</span><span class="s2">&quot;CMake must be installed to build the following extensions: &quot;</span> <span class="o">+</span>
</div><div class="line">                               <span class="s2">&quot;, &quot;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">name</span> <span class="k">for</span> <span class="n">e</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">extensions</span><span class="p">))</span>
</div><div class="line">
</div><div class="line">        <span class="k">if</span> <span class="n">platform</span><span class="o">.</span><span class="n">system</span><span class="p">()</span> <span class="o">==</span> <span class="s2">&quot;Windows&quot;</span><span class="p">:</span>
</div><div class="line">            <span class="n">cmake_version</span> <span class="o">=</span> <span class="n">LooseVersion</span><span class="p">(</span><span class="n">re</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="sa">r</span><span class="s1">&#39;version\s*([\d.]+)&#39;</span><span class="p">,</span> <span class="n">out</span><span class="o">.</span><span class="n">decode</span><span class="p">())</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span>
</div><div class="line">            <span class="k">if</span> <span class="n">cmake_version</span> <span class="o">&lt;</span> <span class="s1">&#39;3.1.0&#39;</span><span class="p">:</span>
</div><div class="line">                <span class="k">raise</span> <span class="ne">RuntimeError</span><span class="p">(</span><span class="s2">&quot;CMake &gt;= 3.1.0 is required on Windows&quot;</span><span class="p">)</span>
</div><div class="line">
</div><div class="line">        <span class="k">for</span> <span class="n">ext</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">extensions</span><span class="p">:</span>
</div><div class="line">            <span class="bp">self</span><span class="o">.</span><span class="n">build_extension</span><span class="p">(</span><span class="n">ext</span><span class="p">)</span>
</div><div class="line">
</div><div class="line">    <span class="k">def</span> <span class="nf">build_extension</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">ext</span><span class="p">):</span>
</div><div class="line">        <span class="n">extdir</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">abspath</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">get_ext_fullpath</span><span class="p">(</span><span class="n">ext</span><span class="o">.</span><span class="n">name</span><span class="p">)))</span>
</div><div class="line">        <span class="c1"># required for auto-detection of auxiliary &quot;native&quot; libs</span>
</div><div class="line">        <span class="k">if</span> <span class="ow">not</span> <span class="n">extdir</span><span class="o">.</span><span class="n">endswith</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">sep</span><span class="p">):</span>
</div><div class="line">            <span class="n">extdir</span> <span class="o">+=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">sep</span>
</div><div class="line">
</div><div class="line">        <span class="n">cmake_args</span> <span class="o">=</span> <span class="p">[</span><span class="s1">&#39;-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=&#39;</span> <span class="o">+</span> <span class="n">extdir</span><span class="p">,</span>
</div><div class="line">                      <span class="s1">&#39;-DPYTHON_EXECUTABLE=&#39;</span> <span class="o">+</span> <span class="n">sys</span><span class="o">.</span><span class="n">executable</span><span class="p">]</span>
</div><div class="line">
</div><div class="line">        <span class="n">cfg</span> <span class="o">=</span> <span class="s1">&#39;Debug&#39;</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">debug</span> <span class="k">else</span> <span class="s1">&#39;Release&#39;</span>
</div><div class="line">        <span class="n">build_args</span> <span class="o">=</span> <span class="p">[</span><span class="s1">&#39;--config&#39;</span><span class="p">,</span> <span class="n">cfg</span><span class="p">]</span>
</div><div class="line">
</div><div class="line">        <span class="k">if</span> <span class="n">platform</span><span class="o">.</span><span class="n">system</span><span class="p">()</span> <span class="o">==</span> <span class="s2">&quot;Windows&quot;</span><span class="p">:</span>
</div><div class="line">            <span class="n">cmake_args</span> <span class="o">+=</span> <span class="p">[</span><span class="s1">&#39;-DCMAKE_LIBRARY_OUTPUT_DIRECTORY_</span><span class="si">{}</span><span class="s1">=</span><span class="si">{}</span><span class="s1">&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">cfg</span><span class="o">.</span><span class="n">upper</span><span class="p">(),</span> <span class="n">extdir</span><span class="p">)]</span>
</div><div class="line">            <span class="k">if</span> <span class="n">sys</span><span class="o">.</span><span class="n">maxsize</span> <span class="o">&gt;</span> <span class="mi">2</span><span class="o">**</span><span class="mi">32</span><span class="p">:</span>
</div><div class="line">                <span class="n">cmake_args</span> <span class="o">+=</span> <span class="p">[</span><span class="s1">&#39;-A&#39;</span><span class="p">,</span> <span class="s1">&#39;x64&#39;</span><span class="p">]</span>
</div><div class="line">            <span class="n">build_args</span> <span class="o">+=</span> <span class="p">[</span><span class="s1">&#39;--&#39;</span><span class="p">,</span> <span class="s1">&#39;/m&#39;</span><span class="p">]</span>
</div><div class="line">        <span class="k">else</span><span class="p">:</span>
</div><div class="line">            <span class="n">cmake_args</span> <span class="o">+=</span> <span class="p">[</span><span class="s1">&#39;-DCMAKE_BUILD_TYPE=&#39;</span> <span class="o">+</span> <span class="n">cfg</span><span class="p">]</span>
</div><div class="line">            <span class="n">build_args</span> <span class="o">+=</span> <span class="p">[</span><span class="s1">&#39;--&#39;</span><span class="p">,</span> <span class="s1">&#39;-j2&#39;</span><span class="p">]</span>
</div><div class="line">
</div><div class="line">        <span class="n">env</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
</div><div class="line">        <span class="n">env</span><span class="p">[</span><span class="s1">&#39;CXXFLAGS&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="s1">&#39;</span><span class="si">{}</span><span class="s1"> -DVERSION_INFO=</span><span class="se">\\</span><span class="s1">&quot;</span><span class="si">{}</span><span class="se">\\</span><span class="s1">&quot;&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">env</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">&#39;CXXFLAGS&#39;</span><span class="p">,</span> <span class="s1">&#39;&#39;</span><span class="p">),</span>
</div><div class="line">                                                              <span class="bp">self</span><span class="o">.</span><span class="n">distribution</span><span class="o">.</span><span class="n">get_version</span><span class="p">())</span>
</div><div class="line">        <span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">build_temp</span><span class="p">):</span>
</div><div class="line">            <span class="n">os</span><span class="o">.</span><span class="n">makedirs</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">build_temp</span><span class="p">)</span>
</div><div class="line">        <span class="n">subprocess</span><span class="o">.</span><span class="n">check_call</span><span class="p">([</span><span class="s1">&#39;cmake&#39;</span><span class="p">,</span> <span class="n">ext</span><span class="o">.</span><span class="n">sourcedir</span><span class="p">]</span> <span class="o">+</span> <span class="n">cmake_args</span><span class="p">,</span> <span class="n">cwd</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">build_temp</span><span class="p">,</span> <span class="n">env</span><span class="o">=</span><span class="n">env</span><span class="p">)</span>
</div><div class="line">        <span class="n">subprocess</span><span class="o">.</span><span class="n">check_call</span><span class="p">([</span><span class="s1">&#39;cmake&#39;</span><span class="p">,</span> <span class="s1">&#39;--build&#39;</span><span class="p">,</span> <span class="s1">&#39;.&#39;</span><span class="p">]</span> <span class="o">+</span> <span class="n">build_args</span><span class="p">,</span> <span class="n">cwd</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">build_temp</span><span class="p">)</span>
</div><div class="line">
</div><div class="line"><span class="n">setup</span><span class="p">(</span>
</div><div class="line">    <span class="n">name</span><span class="o">=</span><span class="s1">&#39;cmake_example&#39;</span><span class="p">,</span>
</div><div class="line">    <span class="n">version</span><span class="o">=</span><span class="s1">&#39;0.0.1&#39;</span><span class="p">,</span>
</div><div class="line">    <span class="n">author</span><span class="o">=</span><span class="s1">&#39;Dean Moldovan&#39;</span><span class="p">,</span>
</div><div class="line">    <span class="n">author_email</span><span class="o">=</span><span class="s1">&#39;dean0x7d@gmail.com&#39;</span><span class="p">,</span>
</div><div class="line">    <span class="n">description</span><span class="o">=</span><span class="s1">&#39;A test project using pybind11 and CMake&#39;</span><span class="p">,</span>
</div><div class="line">    <span class="n">long_description</span><span class="o">=</span><span class="s1">&#39;&#39;</span><span class="p">,</span>
</div><div class="line">    <span class="n">ext_modules</span><span class="o">=</span><span class="p">[</span><span class="n">CMakeExtension</span><span class="p">(</span><span class="s1">&#39;cmake_example&#39;</span><span class="p">)],</span>
</div><div class="line">    <span class="n">cmdclass</span><span class="o">=</span><span class="nb">dict</span><span class="p">(</span><span class="n">build_ext</span><span class="o">=</span><span class="n">CMakeBuild</span><span class="p">),</span>
</div><div class="line">    <span class="n">zip_safe</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</div><div class="line"><span class="p">)</span>
</div></code></pre></div>
</div>
<p>可以看出，它通过重写 <code>setuptools</code> 的 <code>build_ext</code> <code>cmdclass</code> 在构建过程中调用了 <code>cmake</code> 命令完成扩展的构建。</p>
<p>这个方案比较适合 <code>pybind11</code> 的项目，因为它已经提供了很多 CMake 的 module 比如怎么找到 <code>Python.h</code>、<code>libpython</code> 等，打开示例项目的 <code>CMakeLists.txt</code> 可以发现它使用了一个 <code>pybind11</code> 提供的 CMake 函数 <a href="https://pybind11.readthedocs.io/en/stable/compiling.html#pybind11-add-module" title="`pybind11_add_module`"><code>pybind11_add_module</code></a> 来定义 Python 扩展，免去了很多繁琐的配置。</p>
<div class="block-code" data-language="cmake"><div class="highlight"><pre><span></span><code><div class="line"><span class="nb">cmake_minimum_required</span><span class="p">(</span><span class="s">VERSION</span><span class="w"> </span><span class="s">2.8.12</span><span class="p">)</span>
</div><div class="line"><span class="nb">project</span><span class="p">(</span><span class="s">cmake_example</span><span class="p">)</span>
</div><div class="line">
</div><div class="line"><span class="nb">add_subdirectory</span><span class="p">(</span><span class="s">pybind11</span><span class="p">)</span>
</div><div class="line"><span class="nb">pybind11_add_module</span><span class="p">(</span><span class="s">cmake_example</span><span class="w"> </span><span class="s">src/main.cpp</span><span class="p">)</span>
</div></code></pre></div>
</div>
<p>如果不使用 pybind11 则比较麻烦，看看 <a href="https://github.com/apache/arrow/blob/master/python/CMakeLists.txt" title="Apache Arrow Python 包的 `CMakeLists.txt`">Apache Arrow Python 包的 <code>CMakeLists.txt</code></a> 感受一下。</p>
<h2>方案二 scikit-build</h2>
<p>scikit-build 是一个增强的 Python C/C++/Fortan/Cython 扩展构建系统生成器，本质上也是 Python setuptools 和 CMake 的胶水。</p>
<p>我们看一下 sciket-build 的 <a href="https://github.com/scikit-build/scikit-build-sample-projects/tree/master/projects/hello-cpp" title="`hello-cpp`"><code>hello-cpp</code></a> 示例：</p>
<ul>
<li><code>setup.py</code></li>
</ul>
<div class="block-code" data-language="python"><div class="highlight"><pre><span></span><code><div class="line"><span class="kn">import</span> <span class="nn">sys</span>
</div><div class="line">
</div><div class="line"><span class="kn">from</span> <span class="nn">skbuild</span> <span class="kn">import</span> <span class="n">setup</span>
</div><div class="line">
</div><div class="line"><span class="c1"># Require pytest-runner only when running tests</span>
</div><div class="line"><span class="n">pytest_runner</span> <span class="o">=</span> <span class="p">([</span><span class="s1">&#39;pytest-runner&gt;=2.0,&lt;3dev&#39;</span><span class="p">]</span>
</div><div class="line">                 <span class="k">if</span> <span class="nb">any</span><span class="p">(</span><span class="n">arg</span> <span class="ow">in</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span> <span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="p">(</span><span class="s1">&#39;pytest&#39;</span><span class="p">,</span> <span class="s1">&#39;test&#39;</span><span class="p">))</span>
</div><div class="line">                 <span class="k">else</span> <span class="p">[])</span>
</div><div class="line">
</div><div class="line"><span class="n">setup_requires</span> <span class="o">=</span> <span class="n">pytest_runner</span>
</div><div class="line">
</div><div class="line"><span class="n">setup</span><span class="p">(</span>
</div><div class="line">    <span class="n">name</span><span class="o">=</span><span class="s2">&quot;hello-cpp&quot;</span><span class="p">,</span>
</div><div class="line">    <span class="n">version</span><span class="o">=</span><span class="s2">&quot;1.2.3&quot;</span><span class="p">,</span>
</div><div class="line">    <span class="n">description</span><span class="o">=</span><span class="s2">&quot;a minimal example package (cpp version)&quot;</span><span class="p">,</span>
</div><div class="line">    <span class="n">author</span><span class="o">=</span><span class="s1">&#39;The scikit-build team&#39;</span><span class="p">,</span>
</div><div class="line">    <span class="n">license</span><span class="o">=</span><span class="s2">&quot;MIT&quot;</span><span class="p">,</span>
</div><div class="line">    <span class="n">packages</span><span class="o">=</span><span class="p">[</span><span class="s1">&#39;hello&#39;</span><span class="p">],</span>
</div><div class="line">    <span class="n">tests_require</span><span class="o">=</span><span class="p">[</span><span class="s1">&#39;pytest&#39;</span><span class="p">],</span>
</div><div class="line">    <span class="n">setup_requires</span><span class="o">=</span><span class="n">setup_requires</span>
</div><div class="line"><span class="p">)</span>
</div></code></pre></div>
</div>
<p>基本上就是一个 <code>setuptools.setup</code> 的完整替代，不再使用 <code>from setuptools import set</code> 转用 <code>from skbuild import setup</code>。</p>
<ul>
<li><code>CMakeLists.txt</code></li>
</ul>
<div class="block-code" data-language="cmake"><div class="highlight"><pre><span></span><code><div class="line"><span class="nb">cmake_minimum_required</span><span class="p">(</span><span class="s">VERSION</span><span class="w"> </span><span class="s">3.4.0</span><span class="p">)</span>
</div><div class="line">
</div><div class="line"><span class="nb">project</span><span class="p">(</span><span class="s">hello</span><span class="p">)</span>
</div><div class="line">
</div><div class="line"><span class="nb">find_package</span><span class="p">(</span><span class="s">PythonExtensions</span><span class="w"> </span><span class="s">REQUIRED</span><span class="p">)</span>
</div><div class="line">
</div><div class="line"><span class="nb">add_library</span><span class="p">(</span><span class="s">_hello</span><span class="w"> </span><span class="s">MODULE</span><span class="w"> </span><span class="s">hello/_hello.cxx</span><span class="p">)</span>
</div><div class="line"><span class="nb">python_extension_module</span><span class="p">(</span><span class="s">_hello</span><span class="p">)</span>
</div><div class="line"><span class="nb">install</span><span class="p">(</span><span class="s">TARGETS</span><span class="w"> </span><span class="s">_hello</span><span class="w"> </span><span class="s">LIBRARY</span><span class="w"> </span><span class="s">DESTINATION</span><span class="w"> </span><span class="s">hello</span><span class="p">)</span>
</div></code></pre></div>
</div>
<p>这里没有看到类似上面 <code>pybind11</code> CMake 示例中的 <code>add_subdirectory(pybind11)</code> 语句，而是直接用的 <code>find_package(PythonExtensions REQUIRED)</code> 和 <code>python_extension_module</code> CMake 函数：</p>
<ol>
<li><code>PythonExtensions</code> 的 CMake 定义已经<a href="https://github.com/scikit-build/scikit-build/blob/master/skbuild/resources/cmake/FindPythonExtensions.cmake" title="打包在 scikit-build 中">打包在 scikit-build 中</a></li>
<li>调用 <code>skbuild.setup</code> 的过程中 scikit-build 自动把它打包的 CMake 定义文件加载了所以上面才不需要像 <code>pybind11</code> 那样做</li>
<li><code>install(TARGETS _hello LIBRARY DESTINATION hello)</code> 将构建好的扩展的动态链接库复制到 <code>hello/</code> 目录中，从而可以在 Python 中使用 <code>from hello._hello import hello</code> 导入扩展中的 <code>hello</code> 函数</li>
</ol>
<p>通常还会增加 <code>pyproject.toml</code> 来安装 pip 构建时候需要的依赖包：</p>
<div class="block-code" data-language="toml"><div class="highlight"><pre><span></span><code><div class="line"><span class="k">[build-system]</span><span class="w"></span>
</div><div class="line"><span class="n">requires</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="s">&quot;setuptools&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;wheel&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;scikit-build&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;cmake&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;ninja&quot;</span><span class="p">]</span><span class="w"></span>
</div></code></pre></div>
</div>
<p>比较有意思的是，scikit-build 并不需要你的系统上全局安装 CMake/Ninja，它打包了 manylinux 的 <a href="https://pypi.org/project/cmake/" title="CMake 二进制 wheel">CMake</a> 和 <a href="https://pypi.org/project/ninja/" title="Ninja 二进制 wheel">Ninja</a> 的二进制 wheels 并发布到了 PyPi 上，cool.</p>
<p>scikit-build 还支持类似的方式构建使用 <a href="https://github.com/scikit-build/scikit-build-sample-projects/tree/master/projects/hello-cython" title="scikit-build Cython 示例">Cython</a> 和 <a href="https://github.com/scikit-build/scikit-build-sample-projects/tree/master/projects/hello-pybind11" title="scikit-build pybind11 示例">pybind11</a> 等的扩展，功能强大非常方便。</p>
<h2>后记</h2>
<p>最近工作中完成了使用 CMake 和 scikit-build 改造一个 C++ 和 Cython 写的 Python 扩展项目以便能够使用 abseil-cpp 的 Swiss Tables 优化性能，这篇文章差不多就是 brain dump 一下调研的过程，后面我想写一下如何在 Cython 中使用 abseil-cpp 的 containers 的文章，stay tuned.</p>
<hr /><p><a rel="payment" href="https://afdian.com/a/messense">爱发电上赞助</a></p>]]></content:encoded></item><item><title><![CDATA[wechatpy 项目的前世今生]]></title><guid>https://messense.me/wechatpy-project-history</guid><link>https://messense.me/wechatpy-project-history</link><dc:creator><![CDATA[messense]]></dc:creator><pubDate>Sat, 25 Apr 2020 14:26:28 +0000</pubDate><content:encoded><![CDATA[<h2>wechatpy 是什么？</h2>
<p>wechatpy 项目是一个 Python 语言实现的微信公众号开发 SDK，代码托管在 GitHub 上：<a href="https://github.com/wechatpy/wechatpy">https://github.com/wechatpy/wechatpy</a></p>
<h2>wechatpy 前世</h2>
<p>wechatpy 是我 2014 年大学还没毕业在杭州一家使用 Python 开发的创业公司实习的时候创建的项目。当时微信公众号 Python SDK 应该也已经出现了好些个了，一开始公司要做微信公众号开发项目的时候，我选择的是当时网友 <a href="https://github.com/whtsky" title="@whtsky">@whtsky</a> 开发的 <a href="https://github.com/offu/WeRoBot" title="WeRobot">WeRobot</a> 框架（之前也和 @whtsky 合作开发过 <a href="https://github.com/whtsky/Catsup" title="Catsup">Catsup</a> 项目）。</p>
<p>当时使用的 Django 框架，在使用 WeRobot 接入完被动响应功能之后，觉得我想要的微信公众号开发 SDK 更多的是一个库（library）而不是一个框架(framework)，库和框架的主要区别我认为在于：比如实现同一个功能，库是<strong>用户调用库提供的 API</strong>，而框架则是<strong>框架调用用户提供的 API</strong>。就实现微信公众号被动响应功能来说，使用库的方式就已经足够简单了，框架当然也很简单但对维护框架的开发者来说需要为各种 Web 框架再做适配。</p>
<p>而之后接入主动调用接口的时候，因为微信开放的接口数量繁多，WeRobot 把所有的接口都组织在 <code>werobot.Client</code> 类下，我个人不是很喜欢这种做法，并且当时也看了 <a href="https://github.com/nameko/nameko" title="nameko">nameko</a> 的代码，我很喜欢它的那种命名空间的 API 组织方式，便想尝试按自己的想法写一个微信公众号 Python SDK，由此 wechatpy 项目诞生。</p>
<h2>wechatpy 接口设计</h2>
<p>wechatpy 项目初期，复用了 WeRobot 的一些代码，非常感谢 @whtsky 和 WeRobot 项目贡献者（也曾在 WeRobot 项目发起过<a href="https://github.com/offu/WeRoBot/issues/67" title="使用 wechatpy 来做解析消息等工作，WeRoBot 专注 robot 层面的东西">使用 wechatpy 来做解析消息等工作，WeRoBot 专注 robot 层面的东西</a> 的提议，只是被否了）。除此之外，wechatpy 有着几个受到 Django ORM 和 nameko 影响的接口设计。</p>
<h3>一、被动响应消息</h3>
<p>微信被动响应消息的格式是 XML，解析本质上就是把 XML 里面的字段转换成 Python 的 dict，这个使用 <a href="https://github.com/martinblech/xmltodict" title="xmltodict">xmltodict</a> 库可以很方便地实现。但是直接使用 dict 代码中很难知道有哪些字段可以使用，要是有 schema 就好了，当时还没了解过 <a href="https://github.com/marshmallow-code/marshmallow" title="marshmallow">marshmallow</a>，dataclasses 和 <a href="https://github.com/samuelcolvin/pydantic" title="pydantic">pydantic</a> 都还没有面世，基于对 Django ORM model 实现代码的学习，设计了一套类似的机制。</p>
<p>定义了 <code>BaseField</code> 提供 <code>from_xml</code> 和 <code>to_xml</code> 抽象 XML 字段序列化和反序列化，再继承 <code>BaseField</code> 实现具体的字段类型，比如 <code>StringField</code> 代码如下：</p>
<div class="block-code" data-language="python"><div class="highlight"><pre><span></span><code><div class="line"><span class="k">class</span> <span class="nc">StringField</span><span class="p">(</span><span class="n">BaseField</span><span class="p">):</span>
</div><div class="line">    <span class="k">def</span> <span class="nf">__to_text</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
</div><div class="line">        <span class="k">return</span> <span class="n">to_text</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
</div><div class="line">
</div><div class="line">    <span class="n">converter</span> <span class="o">=</span> <span class="n">__to_text</span>
</div><div class="line">
</div><div class="line">    <span class="k">def</span> <span class="nf">to_xml</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
</div><div class="line">        <span class="n">value</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">converter</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
</div><div class="line">        <span class="k">return</span> <span class="sa">f</span><span class="s2">&quot;&lt;</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">name</span><span class="si">}</span><span class="s2">&gt;&lt;![CDATA[</span><span class="si">{</span><span class="n">value</span><span class="si">}</span><span class="s2">]]&gt;&lt;/</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">name</span><span class="si">}</span><span class="s2">&gt;&quot;</span>
</div><div class="line">
</div><div class="line">    <span class="nd">@classmethod</span>
</div><div class="line">    <span class="k">def</span> <span class="nf">from_xml</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
</div><div class="line">        <span class="k">return</span> <span class="n">value</span>
</div></code></pre></div>
</div>
<div class="blockquote"><blockquote><p>f-strings 的用法是最近 <a href="https://github.com/LKI" title="@LKI">@LKI</a> 贡献的，:-)</p>
</blockquote></div>
<p>有了这些字段类型的定义之后，就可以进行消息 model 的定义了，wechatpy 使用 Python 的 metaclass 特性实现了字段类型到数据的序列化和反序列化操作，具体可以参考 <a href="https://github.com/wechatpy/wechatpy/blob/master/wechatpy/messages.py#L35-L72" title="MessageMetaClass">MessageMetaClass</a> 代码。</p>
<p>被动响应消息的回复实现也很类似，在 <code>MessageMetaClass</code> 的基础上，<code>BaseReply</code> 增加了 <code>render()</code> 方法，调用字段的 <code>to_xml</code> 方法将回复 model 序列化为 XML，调用示例：</p>
<div class="block-code" data-language="python"><div class="highlight"><pre><span></span><code><div class="line"><span class="kn">from</span> <span class="nn">wechatpy.replies</span> <span class="kn">import</span> <span class="n">TextReply</span>
</div><div class="line">
</div><div class="line"><span class="n">reply</span> <span class="o">=</span> <span class="n">TextReply</span><span class="p">(</span><span class="n">content</span><span class="o">=</span><span class="s1">&#39;text reply&#39;</span><span class="p">,</span> <span class="n">message</span><span class="o">=</span><span class="n">msg</span><span class="p">)</span>
</div><div class="line"><span class="c1"># 或者</span>
</div><div class="line"><span class="n">reply</span> <span class="o">=</span> <span class="n">TextReply</span><span class="p">(</span><span class="n">message</span><span class="o">=</span><span class="n">msg</span><span class="p">)</span>
</div><div class="line"><span class="n">reply</span><span class="o">.</span><span class="n">content</span> <span class="o">=</span> <span class="s1">&#39;text reply&#39;</span>
</div><div class="line"><span class="c1"># 转换成 XML</span>
</div><div class="line"><span class="n">xml</span> <span class="o">=</span> <span class="n">reply</span><span class="o">.</span><span class="n">render</span><span class="p">()</span>
</div></code></pre></div>
</div>
<h3>二、主动调用接口</h3>
<p>主动调用接口主要是实现了接口的分组，比如主动消息接口都在 <code>WeChatClient.message</code> 实例下，调用方法如下：</p>
<div class="block-code" data-language="python"><div class="highlight"><pre><span></span><code><div class="line"><span class="kn">from</span> <span class="nn">wechatpy</span> <span class="kn">import</span> <span class="n">WeChatClient</span>
</div><div class="line">
</div><div class="line"><span class="n">client</span> <span class="o">=</span> <span class="n">WeChatClient</span><span class="p">(</span><span class="s1">&#39;appid&#39;</span><span class="p">,</span> <span class="s1">&#39;secret&#39;</span><span class="p">)</span>
</div><div class="line"><span class="c1"># 发送图片消息</span>
</div><div class="line"><span class="n">res</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">message</span><span class="o">.</span><span class="n">send_image</span><span class="p">(</span><span class="s1">&#39;openid&#39;</span><span class="p">,</span> <span class="s1">&#39;media_id&#39;</span><span class="p">)</span>
</div><div class="line"><span class="c1"># 查询自定义菜单</span>
</div><div class="line"><span class="n">menu</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">menu</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
</div></code></pre></div>
</div>
<p>这种接口的设计也是基于 Python 的 metaclass 特性实现的，主要代码在 <a href="https://github.com/wechatpy/wechatpy/blob/318643ea412cf116b3bfa3f60fcc29c61d58fe40/wechatpy/client/base.py#L22-L32" title="BaseWeChatClient">BaseWeChatClient</a> 类的 <code>__new__</code> 方法上。</p>
<div class="block-code" data-language="python"><div class="highlight"><pre><span></span><code><div class="line"><span class="kn">import</span> <span class="nn">inspect</span>
</div><div class="line">
</div><div class="line"><span class="kn">from</span> <span class="nn">wechatpy.client.api.base</span> <span class="kn">import</span> <span class="n">BaseWeChatAPI</span>
</div><div class="line">
</div><div class="line"><span class="k">def</span> <span class="nf">_is_api_endpoint</span><span class="p">(</span><span class="n">obj</span><span class="p">):</span>
</div><div class="line">    <span class="k">return</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span> <span class="n">BaseWeChatAPI</span><span class="p">)</span>
</div><div class="line">
</div><div class="line"><span class="k">class</span> <span class="nc">BaseWeChatClient</span><span class="p">:</span>
</div><div class="line">    <span class="k">def</span> <span class="fm">__new__</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
</div><div class="line">        <span class="bp">self</span> <span class="o">=</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__new__</span><span class="p">(</span><span class="bp">cls</span><span class="p">)</span>
</div><div class="line">        <span class="n">api_endpoints</span> <span class="o">=</span> <span class="n">inspect</span><span class="o">.</span><span class="n">getmembers</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">_is_api_endpoint</span><span class="p">)</span>
</div><div class="line">        <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">api</span> <span class="ow">in</span> <span class="n">api_endpoints</span><span class="p">:</span>
</div><div class="line">            <span class="n">api_cls</span> <span class="o">=</span> <span class="nb">type</span><span class="p">(</span><span class="n">api</span><span class="p">)</span>
</div><div class="line">            <span class="n">api</span> <span class="o">=</span> <span class="n">api_cls</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
</div><div class="line">            <span class="nb">setattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">api</span><span class="p">)</span>
</div><div class="line">        <span class="k">return</span> <span class="bp">self</span>
</div></code></pre></div>
</div>
<ol>
<li>使用 <code>inspect.getmembers</code> 获取到类成员中的继承自 <code>BaseWeChatAPI</code> 的实例</li>
<li>通过 <code>type</code> 内置函数获取继承自 <code>BaseWeChatAPI</code> 的实例成员的类型后重新初始化并传入当前 <code>WeChatClient</code> 实例作为构造函数的参数，用 <code>setattr</code> 内置函数替换原来的成员</li>
</ol>
<p>之后需要实现具体的微信主动调用接口，只需要集成 <code>BaseWeChatAPI</code> 并实例为 <code>WeChatClient</code> 的一个成员变量就行了，以菜单接口为例：</p>
<div class="block-code" data-language="python"><div class="highlight"><pre><span></span><code><div class="line"><span class="kn">from</span> <span class="nn">wechatpy.client.base</span> <span class="kn">import</span> <span class="n">BaseWeChatClient</span>
</div><div class="line"><span class="kn">from</span> <span class="nn">wechatpy.client.api.base</span> <span class="kn">import</span> <span class="n">BaseWeChatAPI</span>
</div><div class="line">
</div><div class="line">
</div><div class="line"><span class="k">class</span> <span class="nc">WeChatMenu</span><span class="p">(</span><span class="n">BaseWeChatAPI</span><span class="p">):</span>
</div><div class="line">    <span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
</div><div class="line">        <span class="c1"># 实现代码省略</span>
</div><div class="line">
</div><div class="line">
</div><div class="line"><span class="k">class</span> <span class="nc">WeChatClient</span><span class="p">(</span><span class="n">BaseWeChatClient</span><span class="p">):</span>
</div><div class="line">    <span class="n">API_BASE_URL</span> <span class="o">=</span> <span class="s2">&quot;https://api.weixin.qq.com/cgi-bin/&quot;</span>
</div><div class="line">    
</div><div class="line">    <span class="n">menu</span> <span class="o">=</span> <span class="n">WeChatMenu</span><span class="p">()</span>
</div></code></pre></div>
</div>
<p>就可以愉快地使用 <code>client.menu.get()</code> 方法调用了。</p>
<h2>wechatpy 今生</h2>
<p>wechatpy 项目从创建到今天，已经获得了 2400 多 GitHub 星标，有 600 多 forks，活跃的代码贡献者 3~4 个（感谢！），QQ 群约 500 成员（经常清理不活跃的成员方便新人进群），依然是个比较小的项目。wechatpy 当前支持的微信开放平台功能有：</p>
<ol>
<li>公众号平台被动响应和主动调用 API</li>
<li>企业微信 API</li>
<li>微信支付 API</li>
<li>第三方平台代公众号调用接口 API</li>
<li>小程序和小程序云开发 API</li>
</ol>
<p>也有一些基于 wechatpy 做的周边项目，如：</p>
<ol>
<li><a href="https://github.com/JoneXiong/oejia_wx" title="Odoo 的微信模块">Odoo 的微信模块</a></li>
<li><a href="https://apps.odoo.com/apps/modules/12.0/wechat/" title="Odoo WeChat API">Odoo WeChat API</a></li>
<li><a href="https://github.com/cloverstd/flask-wechatpy" title="flask-wechatpy">flask-wechatpy</a></li>
</ol>
<p>等。</p>
<h2>wechatpy 未来</h2>
<p>目前 wechatpy 项目已经在 2.0 版本的迭代中，最大的改变是放弃了 Python 2 的支持，并使用 <a href="https://github.com/python-poetry/poetry" title="Poetry">Poetry</a> 进行依赖管理。</p>
<p>非常欢迎参与 wechatpy 项目开发的讨论和代码贡献，谢谢。</p>
<hr /><p><a rel="payment" href="https://afdian.com/a/messense">爱发电上赞助</a></p>]]></content:encoded></item><item><title><![CDATA[uWSGI re-fork master 踩坑记]]></title><guid>https://messense.me/uwsgi-re-fork-master</guid><link>https://messense.me/uwsgi-re-fork-master</link><description><![CDATA[Zero downtime Python application deployment]]></description><dc:creator><![CDATA[messense]]></dc:creator><pubDate>Wed, 23 Aug 2017 06:38:39 +0000</pubDate><content:encoded><![CDATA[<h2>0x00 参考文档</h2>
<ol>
<li><a href="http://uwsgi-docs.readthedocs.io/en/latest/articles/TheArtOfGracefulReloading.html">The Art of Graceful Reloading</a></li>
<li><a href="http://zarnovican.github.io/2016/02/15/uwsgi-graceful-reload/">uWSGI graceful Python code deploy</a></li>
</ol>
<p>如上文档选定了实现 zero downtime deployment 方案：re-fork master</p>
<h2>0x01 First Try</h2>
<p>uWSGI 配置文件：</p>
<div class="block-code" data-language="ini"><div class="highlight"><pre><span></span><code><div class="line"><span class="k">[uwsgi]</span><span class="w"></span>
</div><div class="line"><span class="na">chdir</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/home/foo/helloworld</span><span class="w"></span>
</div><div class="line"><span class="na">module</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">helloworld.wsgi</span><span class="w"></span>
</div><div class="line"><span class="na">processes</span><span class="w">  </span><span class="o">=</span><span class="w"> </span><span class="s">4</span><span class="w"></span>
</div><div class="line"><span class="na">socket</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/var/run/uwsgi/app.sock</span><span class="w"></span>
</div><div class="line"><span class="na">pidfile</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/var/run/uwsgi/app.pid</span><span class="w"></span>
</div><div class="line"><span class="na">vacuum</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">false</span><span class="w"></span>
</div><div class="line">
</div><div class="line"><span class="na">master-fifo</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/tmp/app.new.fifo</span><span class="w"></span>
</div><div class="line"><span class="na">master-fifo</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/tmp/app.running.fifo</span><span class="w"></span>
</div><div class="line">
</div><div class="line"><span class="na">if-exists</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/tmp/app.running.fifo</span><span class="w"></span>
</div><div class="line"><span class="w">  </span><span class="na">hook-accepting1-once</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">writefifo:/tmp/app.running.fifo q</span><span class="w"></span>
</div><div class="line"><span class="na">endif</span><span class="w"> </span><span class="o">=</span><span class="w"></span>
</div><div class="line"><span class="na">hook-accepting1-once</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">writefifo:/tmp/app.new.fifo 1P</span><span class="w"></span>
</div></code></pre></div>
</div>
<ol>
<li><p>首先配置了两个 master-fifo，<code>app.new.fifo</code> 为默认，参考 <a href="http://uwsgi-docs.readthedocs.io/en/latest/MasterFIFO.html">The Master FIFO</a></p>
</li>
<li><p>期望的部署情况：</p>
</li>
<li><p>第一次部署时，部署前无任何 uWSGI 进程，此时 <code>app.running.fifo</code> 和 <code>app.new.fifo</code> 均不存在。uWSGI 进程启动，此时 master fifo 为 <code>app.new.fifo</code>，<code>app.running.fifo</code> 文件不存在故 <code>hook-accepting1-once = writefifo:/tmp/app.running.fifo q</code> 不会被执行，<code>hook-accepting1-once = writefifo:/tmp/app.new.fifo 1P</code> 在 uWSGI 第一个 worker 能够接收请求时执行一次将 master fifo 切换成 1 即 <code>app.running.fifo</code> 并更新 pid 文件中 master 进程 pid</p>
</li>
<li><p>再次部署时，使用 <code>echo f &gt; /tmp/app.running.fifo</code> 通知 uWSGI re-fork master，uWSGI re-fork 一组新的 uWSGI 进程，<code>app.running.fifo</code> 已存在，故 <code>hook-accepting1-once = writefifo:/tmp/app.running.fifo q</code> 在 uWSGI 第一个 worker 能够接收请求时执行一次通知旧的 uWSGI 进程优雅退出（即停止接收新的请求并在处理完已经接收的请求后退出），<code>hook-accepting1-once = writefifo:/tmp/app.new.fifo 1P</code> 也执行一次将 master fifo 切换成 1 即 <code>app.running.fifo</code> 并更新 pid 文件中 master 进程 pid</p>
</li>
</ol>
<p>看起来很完美，实际尝试发现：</p>
<ol>
<li>master fifo 文件生成是异步的，第一次部署时执行 <code>hook-accepting1-once = writefifo:/tmp/app.new.fifo 1P</code> 时 <code>app.new.fifo</code> 可能还没有创建，导致启动失败。详细可参考 GitHub issue: <a href="https://github.com/unbit/uwsgi/issues/1486">Zerg dance: writefifo race condition</a></li>
</ol>
<ul>
<li>解决方案上面的 issue 里面已经提到了，使用 <code>spinningfifo</code> 代替 <code>writefifo</code> 即 <code>hook-accepting1-once = spinningfifo:/tmp/app.new.fifo 1P</code>。（spinningfifo 代码已经合并到 uWSGI core 但尚未发布新版本，可以使用 <a href="https://github.com/pennersr/uwsgi-spinningfifo">uwsgi-spinningfifo</a> 作为插件加载）</li>
</ul>
<ol start="2">
<li>解决了第一个问题后，尝试进行第二次部署（即 <code>echo f &gt; /tmp/app.running.fifo</code>），实际发现有时成功有时 echo 会一直阻塞住，<code>lsof | grep fifo</code> 发现 uWSGI 加载的 <code>app.running.fifo</code> 处于删除状态然而 <code>/tmp/app.running.fifo</code> 文件是存在的。GitHub 上搜索发现相关 issue: <a href="https://github.com/unbit/uwsgi/issues/1492">Race condition graceful restart while fork master</a></li>
</ol>
<div class="blockquote"><blockquote><p>After #race_condition_point there is 2 ways:</p>
<p>1. the old master receive 'q' and delete and create fifo-file, after that new master delete and create fifo-file and change master-fifo to 1</p>
<p>2. the new master delete and create fifo file, change master-fifo to 1 and after this old master delete and create fifo file (because we send 'q' him)</p>
</blockquote></div>
<p>显然第二种情况下会导致上述的情况从而导致 echo 阻塞。</p>
<ul>
<li>所以问题的根源在于 old master 和 new master 共用了 <code>app.running.fifo</code> 导致重建 <code>app.running.fifo</code> 时存在竞争条件，上述的 GitHub issue 尚未解决，想到的 workaround 方案很简单，避免共用 <code>app.running.fifo</code>，可以增加一个 <code>app.quit.fifo</code> 专门用来通知旧的 uWSGI master 进程优雅退出</li>
</ul>
<h2>0x02 Second Try</h2>
<p>新的 uWSGI 配置文件：</p>
<div class="block-code" data-language="ini"><div class="highlight"><pre><span></span><code><div class="line"><span class="k">[uwsgi]</span><span class="w"></span>
</div><div class="line"><span class="na">chdir</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/home/foo/helloworld</span><span class="w"></span>
</div><div class="line"><span class="na">module</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">helloworld.wsgi</span><span class="w"></span>
</div><div class="line"><span class="na">processes</span><span class="w">  </span><span class="o">=</span><span class="w"> </span><span class="s">4</span><span class="w"></span>
</div><div class="line"><span class="na">socket</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/var/run/uwsgi/app.sock</span><span class="w"></span>
</div><div class="line"><span class="na">pidfile</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/var/run/uwsgi/app.pid</span><span class="w"></span>
</div><div class="line"><span class="na">vacuum</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">false</span><span class="w"></span>
</div><div class="line">
</div><div class="line"><span class="na">master-fifo</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/tmp/app.new.fifo</span><span class="w"></span>
</div><div class="line"><span class="na">master-fifo</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/tmp/app.running.fifo</span><span class="w"></span>
</div><div class="line"><span class="na">master-fifo</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/tmp/app.quit.fifo</span><span class="w"></span>
</div><div class="line">
</div><div class="line"><span class="na">if-exists</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/tmp/app.running.fifo</span><span class="w"></span>
</div><div class="line"><span class="w">  </span><span class="na">hook-accepting1-once</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">writefifo:/tmp/app.running.fifo 2q</span><span class="w"></span>
</div><div class="line"><span class="na">endif</span><span class="w"> </span><span class="o">=</span><span class="w"></span>
</div><div class="line"><span class="na">hook-accepting1-once</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">spinningfifo:/tmp/app.new.fifo 1P</span><span class="w"></span>
</div></code></pre></div>
</div>
<p>增加了 <code>app.quit.fifo</code>，使用 <code>echo f &gt; /tmp/app.running.fifo</code> 触发部署时，<code>hook-accepting1-once = writefifo:/tmp/app.running.fifo 2q</code> 通知旧的 master 进程切换 master-fifo 到 2 即 <code>app.quit.fifo</code> 后优雅退出（q）</p>
<p>Works great.</p>
<h2>0x03 Caveats</h2>
<ol>
<li>re-fork master 很强大也很危险，如果用到了 uWSGI 的 <code>attach-daemon</code> 等管理后台进程，由于 re-fork master 也会新开启一组后台进程，需要保证这些后台进程可以启动多个而没有副作用。</li>
<li><code>uwsgi --build-plugin</code> 如果没有任何输出，应尝试安装缺失的依赖如 <code>python</code>, <code>gcc</code> 等。</li>
</ol>
<hr /><p><a rel="payment" href="https://afdian.com/a/messense">爱发电上赞助</a></p>]]></content:encoded></item><item><title><![CDATA[Apple TV 国内观看 Netflix/HBO]]></title><guid>https://messense.me/apple-tv-netflix-hbo</guid><link>https://messense.me/apple-tv-netflix-hbo</link><description><![CDATA[Setup shadowsocks and dnsmasq on OpenWrt for Apple TV]]></description><dc:creator><![CDATA[messense]]></dc:creator><pubDate>Sun, 09 Jul 2017 11:54:15 +0000</pubDate><content:encoded><![CDATA[<p>如果你不愿意折腾的话，也可以尝试 <a href="https://www.getflix.com.au">Getflix</a> 提供的 Smart DNS 解决方案，只需要在 Apple TV 上手动配置 DNS 即可，缺点是并不稳定且限制固定 IP 使用、对 Netflix 的支持比较复杂。或者试试这里  <a href="http://comparitech.net/netflix-vpn">http://comparitech.net/netflix-vpn</a> 列出来的支持 Netflix 的 VPN.</p>
<h2>设备</h2>
<ol>
<li>Apple TV</li>
<li>支持 OpenWrt 的路由器</li>
</ol>
<h2>配置 shadowsocks</h2>
<p><strong>首先，你需要一个支持 Netflix/HBO 等地区解锁的 shadowsocks 服务</strong>，我目前用的是 <a href="https://xeton.io/aff.php?aff=245">rixCloud</a> 提供的 Relay 服务，需要在路由器上安装配置 shadowsocks。</p>
<p>我使用的是小米路由器 mini，直接用了 <a href="https://github.com/blademainer/miwifi-ss">miwifi-ss</a> 的方案按照配置了 shadowsocks ss-redir 服务。</p>
<div class="block-code" data-language="bash"><div class="highlight"><pre><span></span><code><div class="line"><span class="nb">cd</span> /tmp
</div><div class="line">curl https://raw.githubusercontent.com/blademainer/miwifi-ss/master/miwifi.sh -o miwifi.sh
</div><div class="line">chmod +x miwifi.sh
</div><div class="line">sh ./miwifi.sh
</div><div class="line">rm miwifi.sh
</div></code></pre></div>
</div>
<h2>配置 iptables</h2>
<p><a href="https://github.com/blademainer/miwifi-ss">miwifi-ss</a> 中提供的 iptables 配置太过简单，由于 Netflix/HBO 等服务均会验证用户 IP 位置，我们需要自行配置 iptables 规则：</p>
<p><code>/etc/firewall.user</code>:</p>
<div class="block-code" data-language="text"><div class="highlight"><pre><span></span><code><div class="line">iptables -t nat -N SHADOWSOCKS
</div><div class="line">
</div><div class="line">iptables -t nat -A SHADOWSOCKS -d YOUR_SS_SERVER_IP -j RETURN
</div><div class="line">
</div><div class="line">iptables -t nat -A SHADOWSOCKS -d 0.0.0.0/8 -j RETURN
</div><div class="line">iptables -t nat -A SHADOWSOCKS -d 10.0.0.0/8 -j RETURN
</div><div class="line">iptables -t nat -A SHADOWSOCKS -d 127.0.0.0/8 -j RETURN
</div><div class="line">iptables -t nat -A SHADOWSOCKS -d 169.254.0.0/16 -j RETURN
</div><div class="line">iptables -t nat -A SHADOWSOCKS -d 172.16.0.0/12 -j RETURN
</div><div class="line">iptables -t nat -A SHADOWSOCKS -d 192.168.0.0/16 -j RETURN
</div><div class="line">iptables -t nat -A SHADOWSOCKS -d 224.0.0.0/4 -j RETURN
</div><div class="line">iptables -t nat -A SHADOWSOCKS -d 240.0.0.0/4 -j RETURN
</div><div class="line">
</div><div class="line">iptables -t nat -A SHADOWSOCKS -p tcp -j REDIRECT --to-ports 1081
</div><div class="line">
</div><div class="line">iptables -t nat -A PREROUTING -p tcp -j SHADOWSOCKS
</div></code></pre></div>
</div>
<p>配置好后重启 firewall 服务：</p>
<div class="block-code" data-language="bash"><div class="highlight"><pre><span></span><code><div class="line">/etc/init.d/firewall restart
</div></code></pre></div>
</div>
<h2>DNS</h2>
<p>即便配置好了 shadowsocks 和 iptables，仍然很有可能会遇到 DNS 投毒的情况，建议配置下 dnsmasq，可以使用 [gfwlist2dnsmasq][gfwlist2dnsmasq] 生成 dnsmasq 配置文件并放到 <code>/etc/dnsmasq.d</code> 文件夹中，然后重启 dnsmasq 服务：</p>
<div class="block-code" data-language="bash"><div class="highlight"><pre><span></span><code><div class="line">/etc/init.d/dnsmasq restart
</div></code></pre></div>
</div>
<p>我目前使用的 <code>gfwlist.conf</code> 内容如下：</p>
<p><a href="https://gist.github.com/messense/beaea80ba3e3f929e8f71bfe1b383e83">https://gist.github.com/messense/beaea80ba3e3f929e8f71bfe1b383e83</a></p>
<p>Now enjoy some nice TV shows. :-)</p>
<p>[gfwlist2dnsmasq]: <a href="https://github.com/cokebar/gfwlist2dnsmasq">https://github.com/cokebar/gfwlist2dnsmasq</a></p>
<hr /><p><a rel="payment" href="https://afdian.com/a/messense">爱发电上赞助</a></p>]]></content:encoded></item></channel></rss>